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The World Health Organization was established in 1948 as a spe- 
cialized agency of the United Nations serving as the directing and 
coordinating authority for international health matters and public 
health. One of WHO’s constitutional functions is to provide objec- 
tive and reliable information and advice in the field of human health, 
a responsibility that it fulfils in part through its publications pro- 
grammes. Through its publications, the Organization seeks to sup- 
port national health strategies and address the most pressing public 
health concerns. 

The WHO Regional Office for Europe is one of six regional 
offices throughout the world, each with its own programme geared 
to the particular health problems of the countries it serves. The 
European Region embraces some 870 million people living in an 
area stretching from Greenland in the north and the Mediterranean 
in the south to the Pacific shores of the Russian Federation. The 
European programme of WHO therefore concentrates both on the 
problems associated with industrial and post-industrial society and 
on those faced by the emerging democracies of central and eastern 
Europe and the former USSR. 

To ensure the widest possible availability of authoritative infor- 
mation and guidance on health matters, WHO secures broad inter- 
national distribution of its publications and encourages their transla- 
tion and adaptation. By helping to promote and protect health and 
prevent and control disease, WHO’s books contribute to achieving 
the Organization’s principal objective — the attainment by all people 
of the highest possible level of health. 
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Foreword 


For over a decade, the Member States of the WHO European Region 
have been demonstrating their commitment to improving environ- 
mental health services. The First European Conference on Environ- 
ment and Health, held in Frankfurt-am-Main in 1989, spelled out 
the principles for integrating environmental health services. Five years 
later, the Second European Conference in Helsinki provided a clear 
mandate for action on environment and health in Europe through 
the development of national environmental health action plans 
(NEHAPS). The need for improved policy tools to strengthen the 
management of environmental health services was also emphasized 
in Helsinki. Subsequently, at the Third European Conference, in Lon- 
don in 1999, even further attention was paid to local processes for 
planning, implementing and evaluating environment and health ac- 
tivities. 


NEHAPS have become a tool that provide Member States with a 
substantive framework for developing a path towards sound envi- 
ronment and health management, in partnership with all sectors. 
Many Member States have also asked for guidance on specific as- 
pects of the delivery of environmental health services. This series of 
publications entitled Environmental Health Services in Europe was 
developed to respond to these requests. The first two volumes out- 
line the concepts applicable to environmental health and environ- 
mental health services and describe the current status of environ- 
mental health practice in the European Region. The third and fourth 
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volumes address staffing and professional profiles, and provide guid- 
ance on the education and training of environmental health profes- 
sionals. 


This, the fifth in the series, provides practical guidelines — re- 
peatedly requested by the Member States — for evaluating environ- 
mental health services. Evaluation is an essential component of all 
health service delivery that is too often overlooked. It is frequently 
ignored because it is not always seen as directly productive. Never- 
theless, evaluation is the only way to learn from the successes and 
failures of the past and thereby improve policies and procedures for 
the future. To illustrate the difficult concept of evaluation, various 
examples of environmental health service evaluation are presented. 


The production of this book involved a series of discussions and 
meetings with a wide range of dedicated and knowledgeable indi- 
viduals and representatives of Member States. In particular, I would 
like to thank Ms Drew and Mr van Duivenboden, without whose 
persistence and commitment the book could not have been com- 
pleted. In addition, the many consultants and advisers who have 
worked on this project must be thanked for their gracious donation 
of time and expertise. Specifically, Mr Martin Eriksson and 
Dr Andreas Kappos were instrumental in the development of the case 
studies presented in Chapter 6, and Ms Isabelle Goi, Mr Jean Pierre 
Foirry and Mr Laurent Gilotte contributed significantly to the sec- 
tion on economic evaluation tools in Chapter 4. 


It is hoped that this volume will lead to a better understanding of 
evaluation, to new initiatives to introduce evaluation into daily prac- 
tice and to more discussion of evaluation among environmental health 
service providers. 


Marc Danzon 
WHO Regional Director for Europe 


Introduction 


The aim of this book is to provide practical guidelines that assist 
local environmental health practitioners in evaluating their services. 
Because the productive worth of evaluation is often obscure, this 
management step is frequently overlooked. Nevertheless, it is an 
essential component in developing, implementing and improving 
the policies designed to protect health and the environment and en- 
hance the quality of life. Without assessing existing services, their 
intended consequences will remain unknown, as will knowledge of 
whether the proper instruments and timing have been employed. In 
a climate where environmental health services are challenged by the 
sheer magnitude and diversity of services, limited resources and 
many scientific uncertainties, evaluation is especially critical. It is 
the only way to learn the benefits and cost to society of environmen- 
tal services. 


The management cycle includes policy development, implemen- 
tation and the crucial step of evaluation. Because the broad range of 
environmental health services demands different evaluation strate- 
gies, and because the different evaluation concepts, tools and com- 
munication techniques can be combined in an infinite number of 
ways, no single or ideal evaluation model can be offered. The goal 
of this publication is, therefore, to introduce environmental health 
professionals to evaluation principles, tools and examples from which 
they can pick and choose elements to suit their specific needs. It is 
designed not to answer all the questions practitioners have, but rather 
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to guide them towards questions that must be answered to evaluate 
and improve services. 


This volume has evolved from several interactions with Member 
States of the European Region that have identified gaps in the man- 
agement and evaluation of environmental health ser-vices. The three 
European conferences on environment and health — held in Frank- 
furt-am-Main, Germany (1989), Helsinki, Finland (1994) and Lon- 
don, United Kingdom (1999) — all emphasized the need for inte- 
grating environmental health services, developing policy tools, and 
strengthening local processes for planning, implementing and evalu- 
ating environment and health activities (1—5). In addition, previous 
volumes in the series on environmental health services in Europe 
have identified the lack of a clear evaluation methodology (6,7) and 
emphasized the need for environmental health service managers to 
obtain evaluation skills (8,9). The need to improve evaluation meth- 
odology was specifically endorsed by the Second Consultation on 
Environmental Health Services, held in Vilnius, Lithuania, in No- 
vember 1994 (10). 


The structure and format of the present publication are the result 
of a comprehensive study of evaluation methodology, a review meet- 
ing with several environmental health professionals and the conclu- 
sions (71) and recommendations of the Second Consultation of Sen- 
ior Government Officials on the Development of National Environ- 
mental Health Action Plans, held in Dubrovnik, Croatia, in October 
1996 (12). 


The elements discussed and proposed in this book should be used 
by environmental health professionals (at international, national and 
local administrative levels) to assist in a number of management 
activities, such as: 


e improving the quality and quantity of information reaching 
managers, thereby strengthening their ability to make deci- 
sions about future environmental health services; 


e choosing among and prioritizing the various types of envi- 
ronmental health service; 
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* accounting for expenditures on services to political leaders 
and communities; 


e assessing whether policies have been effective in achieving 
their objectives; 


e identifying opportunities for more efficient delivery of envi- 
ronmental health services; 


e providing a mechanism for recognizing changes in public 
needs and wants, thereby allowing environmental health ser- 
vices to be adjusted accordingly; 


e identifying more specific research needs; and 
e interacting with the community to ascertain its wants and needs. 


This book is organized into six chapters. Chapter 1 lays the 
foundation for the rest of the book. It reviews evaluation princi- 
ples and focuses on issues pertaining to preparation, planning 
and design. 


Chapter 2 reviews the concepts that underlie environmental health 
services and discusses how they relate to evaluation. It reviews the 
development of environmental health services and the inherent dif- 
ficulties in evaluating these services. The objectives of, potential 
subjects of and partners for evaluation are also discussed. 


Chapter 3 discusses in detail data and indicators, the essential 
building blocks of evaluation. It describes characteristics that make 
indicators most useful and discusses in some detail several types of 
indicator (including process indicators, environmental indicators, 
health indicators and urban indicators). 


Chapter 4 focuses on the process of conducting an evaluation. It 
presents methods for collecting and analysing the information re- 
quired for evaluation. Particular attention is given to two areas of 
environmental health service evaluation that have thus far been over- 
looked: economic evaluation tools and qualitative evaluation tech- 
niques. 
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Chapter 5 focuses on the final stages of evaluation: writing and 
disseminating results and conclusions in ways that foster their use in 
management decisions. It reviews potential uses for evaluation and 
discusses factors contributing to effective use. 


Chapter 6 presents several case studies of evaluation. 


The need for evaluation is well documented, as are many strate- 
gies for its execution. The purpose of this book is not to repeat exist- 
ing efforts, but to apply evaluation concepts specifically to environ- 
mental health services. With this objective in mind, references and 
suggestions for further reading appear at the end of each chapter, 
and key recommendations are presented in boxes throughout the text. 
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Evaluation Principles 
and Procedures 


This chapter reviews the critical components and purposes 
of evaluation, as well as general theories of evaluation. Ques- 
tions discussed here include: Why evaluate? Who evaluates? 
What can be evaluated? and What evaluation criteria can be 
used? In addition, many of the terms used throughout the 
book are defined. Finally, this chapter presents a general 
evaluation plan, which lays a foundation for the rest of the 
book. 


Evaluation of health services is an established field. Several books 
and reports have been published on the subject, both inside and 
outside of WHO (1—3). This publication does not intend to rewrite 
these documents or restate their arguments. Instead, it aims to fill a 
gap identified by Member States: to provide a practical introduction 
for environmental health professionals and managers to evaluate their 
services. The first task 1s to review the most critical components and 
aim of an evaluation, to see how these relate to the overall frame- 
work of environmental health service management. 


The WHO Regional Office for Europe has developed a strategy 
for promoting health for all in the twenty-first century. Goals and 
objectives of this plan are broken down into 21 targets (4). These 
targets not only define the health goals for the European Region, 
but also describe actions and initiatives that can be taken by Mem- 
ber States to reach the targets. Health for all monitoring reports such 
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as the one published in 1998 (5) present the latest health statistics 
for the European Region. Reporting progress towards the targets is a 
key component of the health for all evaluation strategy. These re- 
ports, however, represent an extremely broad effort to evaluate health 
programmes at the regional level. Individual Member States are also 
responsible for monitoring and evaluating their health services. 


WHAT IS EVALUATION? 


Over the years many definitions of evaluation have been developed 
(1—3,6,7). In 1981 WHO defined evaluation as, “A systematic way 
of learning from experience and using the lessons learned to im- 
prove current activities and promote better planning by careful se- 
lection of alternatives for future action” (1). Another definition of 
policy evaluation is: “A systematic assessment and appraisal of policy 
content, organization, process and effects” (6). These definitions 
provide a foundation for action, but they do not fully describe the 
contribution of evaluation to the overall management process. 


Evaluation is one aspect of the broad management strategy sought 
by WHO and its Member States. Fig. 1 shows a simplified concep- 
tion of this management “cycle”. Policies are first developed and 
then implemented. Thereafter, evaluation assesses the quality of vari- 
ous aspects of those policies. Policy effects are not the sole focus of 
evaluation because other aspects of policy implementation and de- 
velopment may also be covered. Furthermore, even before a policy 
is implemented, evaluation may occasionally contribute to the man- 
agement process. Moreover, identifying and appraising alternative 
actions can be one of the contributions of evaluation. Finally, the 
cycle is completed by taking experience and knowledge gained 
through the evaluation and feeding them back into the policy devel- 
opment phase. This feedback is an integral part of the management 
process. If the data collected are ignored in formulating future poli- 
cies, then valuable resources are wasted. To guide future planning 
and decision-making, evaluation should be both a flexible and action- 
oriented part of the management cycle. 


Comparisons are a necessary part of evaluation. They provide a 
framework for analysis; a given situation is often more fully understood 
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Fig. 1. Management cycle 


Policy formulation 


Evaluation 


Implementation 





if it is compared with something else. Generally, evaluators make 
one of three possible types of comparison: the present situation is 
compared with the past situation, with other situations (similar in 
context) or with a predetermined standard based on statistical, eco- 
nomic, political or social theories. 


Fig. 2 illustrates the three types of comparison. The dashed line 
represents planned progress of a particular programme, the dotted 
line represents actual progress and the solid line represents the 
progress of another programme. 


Fig. 2. Types of comparison used in evaluation 
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Looking at point X in isolation presents little useful information 
to a manager. However, when point X is compared to points A, B or 
C (that is, to the past situation, to another situation or to the planned 
targets), it is possible to judge the speed and progress of the pro- 
gramme — in other words, to evaluate it. Comparing data from the 
present situation to historical data (point X to point A) is one of the 
most common comparisons, and rightly so. It is important to under- 
stand the trends (direction) associated with the progress of a pro- 
gramme. Comparing point X to another programme (point B) may 
be useful as well, as long as circumstances support a fair compari- 
son. The comparison of actual to planned progress (points X and C) 
is also possible. In some cases, however, there is no specific or op- 
erational target with which to compare point X. 


Time is an extremely important element in all evaluations; ide- 
ally, comparisons should be made more than once. One-time analy- 
ses provide only limited “snapshots” of a given situation. To judge 
the progress of an activity over time, a whole series of snapshots is 
preferable. If the time scale is long enough, several evaluations can 
be conducted at regular intervals, and a monitoring system would 
provide information for the interim periods. If a project’s time scale 
is shorter, the frequency of such snapshots should be adjusted. 


DESIGNING AND PREPARING FOR EVALUATIONS 


Evaluation is complex and requires time, resources and planning. 
This section will address the overall evaluation design and several 
factors that contribute to the success of an evaluation. Additional 
details are also provided in Chapter 5. 


Designing evaluations 

Before the individual components of an evaluation are addressed in 
detail, the overall design must be understood. First, a manager should 
decide on the general scale of the evaluation — large, small, or some- 
where in between. It may be helpful to remember that the magnitude 
of an evaluation should correspond to its purpose and that all the 
elements will generally have the same scale. Fig. 3 portrays (in an 
array) some of the different characteristics of an evaluation. Each 
characteristic is discussed further in subsequent sections of this chapter. 
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Fig. 3. Scale and characteristics of evaluation 
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The characteristics vary in scale. In general the level of intensity for 
all the different characteristics will be relatively similar. 


Examples that illustrate the extreme cases are relatively easy to 
find. A small-scale evaluation might be a 15-minute daily staff meet- 
ing intended to provide an ongoing channel of communication within 
a certain group of people. This is an inexpensive activity, adding 
little to the overall cost of a project. Only a few people are involved, 
with no specific skill mix, and the intended audience would be the 
managers themselves. Little planning is required, but a good leader 
might be needed to keep discussions relevant and productive. Pub- 
lic participation would be limited given the frequency and detailed 
nature of this type of evaluation. A clear advantage to this type of 
evaluation is its ability to quickly and inexpensively provide infor- 
mation about day-to-day operations. Such an evaluation is limited, 
however, because of the tendency to focus on small day-to-day prob- 
lems rather than on larger institutional or structural problems. 


In contrast, a large-scale formal evaluation can be appropriate 
for a service funded by a national or international grant programme. 
Such an evaluation might take place after a programme has been in 
place for several years, and may focus on the efficiency of the ser- 
vice or on the outcome in the population. It may take several days or 
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weeks to complete, and might be organized during major project 
milestones only (for example, the midpoint or end). Personnel re- 
quirements are likely to be greater owing to the need for qualitative 
and quantitative (including statistical) analysis and owing to the 
broader range of issues that are likely to be evaluated. To provide 
evaluators with the information required, collaboration with a broad 
base of programme staff is necessary. Systematic planning and 
financial resources are also required in this type of evaluation. A 
formal report is likely to be written and disseminated to interested 
parties, including the granting organization. The advantage of this 
type of evaluation is that it is a thorough, fundamental analysis rather 
than an incremental improvement. But this also implies that small- 
scale developments within a service may go unnoticed, especially 
when evaluators are not from the programme under study. 


Fig. 3 is a simplified presentation, and most evaluations will fall 
somewhere between the two extreme examples just described. 
Because the scales will not always be exactly proportional and 
because straight lines cannot always be drawn across the various 
characteristics displayed in Fig. 3, flexibility has to be maintained, 
and managers need to keep in mind how the information collected 
for the evaluation will be used to help them improve services. 


Preparing for evaluations 

Preparing for an evaluation requires consideration of several factors 
that affect its success. These factors include support, flexibility, cost 
and public participation. Each factor is discussed in more detail be- 
low. 


Support 

Appropriate financial and political support is essential for effective 
evaluation, as is the support of staff and management. For large- 
scale evaluations, broad support at both the national and the local 
level helps provide an atmosphere that is more conducive to pro- 
ducing meaningful results. For smaller-scale evaluations, such as 
the daily 15-minute staff meeting, support is manifested in other 
ways, such as staff punctuality, germane discussions and the intro- 
duction of new ideas that improve service. In both large-scale and 
small-scale evaluations, a sound communication strategy is essential 
to secure political support — before, during and after an evaluation. 
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Flexibility and flow of information 

Evaluations have to be responsive to unforeseen situations that arise. 
For time-consuming, large-scale evaluations in particular, the strat- 
egy should be flexible enough to deal with changes in context. One 
key to maintaining flexibility is information flow, the foundation of 
the management process. Essential to information flow is the selec- 
tion and delivery of important information to the right person at the 
right time. Because an evaluation is largely a matter of collecting 
data, gathering information and feeding these back into the deci- 
sion-making process, it is important to integrate the evaluation with 
the information system. Again, the exchange of information is the 
key issue here. Medium- to large-scale evaluations, therefore, should 
follow a similar process, consisting of codified steps defined before 
performing evaluation activities. In smaller-scale projects and on- 
going evaluations, some of the steps are less formal or can be com- 
bined. To ensure flexibility and the flow of information, every envi- 
ronmental health service activity should establish evaluation as a 
component at the outset of that activity. Evaluation plans can and 
should contribute significantly to effective, efficient and equitable 
environmental health service management. 


Cost 

The cost of an evaluation is also an important consideration. Each 
step in the evaluation process (outlined below) is associated with a 
specific cost, either direct or indirect. Direct costs, such as actual 
resources needed for the evaluation team (supplies, salaries, office 
space, time) are easier to account for than indirect costs (which are 
not identifiable with a specific product, function or activity). The 
cost of not completing an alternative task — the opportunity cost — 
should also be considered. For the planning process to be effective, 
proper financing, time and other resources have to be set aside to 
perform the evaluation, as well as to produce, copy and disseminate 
results. Cost issues and economic tools are discussed in more detail 
in Chapter 4. 


Public participation and openness 

The importance of public participation, open decision-making and 
access to information has steadily gained recognition throughout 
the European Region. Agenda 21 and the Rio Declaration empha- 
sized the need for public participation in achieving sustainable 
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development. The European Charter on Environment and Health, 
adopted in Frankfurt-am-Main, Germany, in 1989, highlighted the 
importance of the public in dealing with environmental and health 
challenges. In Helsinki, Finland, in 1994, the public and 
nongovernmental organizations (NGOs) were recognized as indis- 
pensable partners in the development of national environmental 
health action plans (NEHAPS). The most significant development 
in this area occurred at the Fourth Ministerial Conference “Environ- 
ment for Europe” held in Arhus, Denmark, in 1998 (8). At this land- 
mark event, the participants endorsed the Convention on Access to 
Information, Public Participation in Decision-making and Access to 
Justice in Environmental Matters, which provided a legal framework 
for environmental civic rights in the Region. 


Building on the Arhus Convention was a major topic of discus- 
sion at the Third Ministerial Conference on Environment and Health, 
held in London, United Kingdom, in 1999. The following are some 
examples of important issues that relate to access to information, 
public participation and access to justice that were highlighted at the 
Conference (9). 


e Access to information is not only critical for making sound 
public decisions that relate to environment and health policy, 
but is also critical for individuals, so that they can make in- 
formed choices to improve their own health and environment. 


¢ Information policy should aim to ensure that public authori- 
ties are accountable and operate transparently, and that the 
public is better informed through awareness-raising measures. 


e A system that meets demands for public access to information 
should contain a general presumption in favour of access, de- 
lineate the scope of information covered and the range of bod- 
ies required to provide it, broadly define the terms of access 
(time limits, costs, form), and make provisions for a limited 
number of exemptions. 


e Individuals should always have the right to access informa- 
tion about their own health. Individual privacy should be re- 
spected by structuring health information to prevent identifi- 
cation of individuals. 
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* Information held by the private sector should be addressed to 
ensure its adequate flow into the public domain. 


e Public communication and active dissemination of informa- 
tion are essential for developing and implementing environ- 
mental health policies. A well informed public is more able to 
participate effectively in decision-making and more likely to 
support policies designed to create a healthier environment. 


e Communication on environmental and health matters is not a 
simple one-way process but should involve debate, dialogue 
and feedback, especially during decision-making processes. 


e Several key “actors”, including governments, doctors, envi- 
ronmental health professionals, educational institutions, busi- 
nesses, trade unions and NGOs, play important communica- 
tion roles. 


e Communication technologies are rapidly evolving and should 
be incorporated into communication policies. Television, ra- 
dio, newspapers, the Internet, educational institutions and prod- 
uct labelling are just some of the important ways to provide 
information to the public. New technologies for expanding 
the dialogue with the public are also evolving. It must be rec- 
ognized, however, that new technologies bring new problems 
as well as new opportunities. 


¢ The Århus Convention provides a legal framework for public 
participation in specific activities: programmes, plans and poli- 
cies (Article 6), and general rules and regulations (Articles 7 
and 8, respectively). 


e Access to the judicial system is fundamental to providing the 
public with meaningful involvement in environmental and 
health matters. Efforts should be made to overcome financial 
and practical barriers to such access. 


In accordance with the principles laid out in Arhus and London, 
evaluations should occur in an open and interactive environment 
that includes dialogue with anyone affected by or interested in the 
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activities under review (stakeholders).' Public involvement will not 
always be feasible; certain evaluations may be too small in scope 
and too detailed for extensive public interaction. Open and trans- 
parent communication, however, is always warranted and, in most 
cases, evaluation results should be made publicly available. 


THE EVALUATION PLAN 


Before embarking on any kind of evaluation, the development of a 
clear plan is necessary. This plan informs all parties involved about 
the scope, purpose, methodology, information base and intended 
use of the results. The plan should clearly state what data are to be 
collected. Since many evaluations are based on comparisons over 
time of the same data, managers have to ensure that the correct data 
are collected from the outset. A clear plan also helps to maintain 
fairness in evaluations. The plan should clearly state what criteria 
will be used for the evaluation. This is particularly important when 
evaluators are drawn from outside the programme or process to be 
evaluated. A clear plan can also avoid resistance to an evaluation. 
Many people do not like to be evaluated because they feel they are 
being judged under circumstances they cannot control. If managers 
know at the outset what criteria will be used to judge them, they can 
act to improve these elements. In addition, criteria have to be chosen 
and defined very carefully, because there is a risk that service ac- 
tivities will only focus on the criteria that are measured specifically. 
Whenever possible, the plan should be developed and agreed to in 
writing. Finally, to remain responsive, the plan should be flexible 
and allow for adjustments that might become necessary during the 
evaluation process. The steps presented in Fig. 4 provide a broad 
guide that can be tailored to, and thus used in, almost any type of 
evaluation. 


Fig. 4 reflects the most important elements of virtually all evalu- 
ations. The factors (or conditions) that foster effective evaluations 
include sufficient support, sound communication, openness and flex- 
ibility. These factors are discussed below. 





! Throughout this volume, the term “stakeholder” is broadly defined as parties inter- 
ested in or affected by environmental health services. 


EVALUATION PRINCIPLES AND PROCEDURES 17 





Fig. 4. Suggested steps for an evaluation plan 
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The first major step in an evaluation plan is to secure operational, 
political and financial support. In some cases (as indicated by dotted 
arrows in Fig. 4), the later phases of an evaluation may also influ- 
ence support. For example, preliminary results of an evaluation may 
encourage managers to support a more elaborate dissemination strat- 
egy. The “audience”, the people for whom the evaluation takes place, 
can also influence the amount of support an evaluation will get along 
the way, though sometimes only indirectly. 


The second major step is the development of an evaluation de- 
sign. This involves the identification of the purpose, scope, subject, 
frequency and timing of the evaluation. It is also recommended that 
before carrying out the evaluation, the design should account for 
the dissemination of the evaluation results. Though the evaluation 
design is partially the responsibility of the evaluator, it may also be 
the work of the initiators of the evaluation. 
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The third major step is to perform the evaluation itself. This step 
involves three basic activities. First, identify and select one or more 
suitable methods for gathering and analysing the necessary data. 
Evaluators should consider using both quantitative and qualitative 
tools for gathering data. A combination of these is recommended 
(see also Chapter 4). Second, gather data and use them to determine 
such components as service effectiveness, efficiency and outcome. 
Last, develop and document the results and conclusions. 


The fourth major step is to disseminate the results to the appro- 
priate audience. Keep in mind that the steps and activities in Fig. 4 
comprise only a very rough outline of a real evaluation. (A more 
complete sample outline for an evaluation plan is presented in Box 1.) 
In addition, it is important to remember that the steps described should 
be closely integrated into the management cycle. Evaluation is not a 
separate or isolated process, but an essential element of formulat- 
ing, implementing and improving environmental health services. 


THE PURPOSE: WHY EVALUATE? 


There are many reasons for evaluating services. In a health service 
context, often the broad purpose is to “improve health programmes 
and the health infrastructure for delivering them, and to guide allo- 
cation of resources in current and future programmes” (9). In individual 
situations, many different questions about a particular service may be 
answered by the evaluation. The following is a list of such questions. 


e What has been achieved? 

e Are the objectives being fulfilled? 

e Could the objectives be fulfilled more cheaply? 

e Are the programmes and practices fair and are costs and ben- 
efits equitably distributed? 

e Are the activities sustainable for future generations? 

e What difference has the service made? 

e Are the objectives correct and sufficient for this population? 

e Are the needs of the population being met? 

e What are the strengths and weaknesses of the service? 

e Could the resources used be put to better use by providing 
different services? 
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Box 1. Evaluation plan outline 





Preface: Approaches to foster support, openness, good communication and 
flexibility 


l. Support 


A. Organization providing funding 
B. Expected support from management and appropriate political officials 
C. Target audience and other stakeholders 


Il. Design 


Purpose 

Scope 

Subject 

Frequency 

Timing 

Cost and benefits 

Public participation 

Evaluation personnel 

e required skill mix for personnel 
e number personnel needed 

|. Dissemination strategy, audience 


IONO 


Ill. Actual evaluation 


A. Methodology 
e data to be collected 
e indicators 
e methods of analysis 
B. Results 
C. Conclusions and recommendations 
e linking evaluation to future plans and policies 
e strategy for maximizing use of results 


IV. Dissemination of results 


A. Audience(s) 
B. Method(s) of dissemination 
C. Format of expected product 





e Is the amount of information collected appropriate? 

e What can be done to improve efficiency? 

e How can this information be used to plan future services? 

e What are the implications for future service levels? 

e How should current policies be adapted in response to the 
new knowledge? 


Managers have to ask themselves why questions such as those 
above should be answered. Many of the questions are narrowly 
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focused, but in combination these questions may lead to a greater 
understanding of the overall context. Though economic factors may 
often be the primary motivation for starting an evaluation, this is not 
always the case. For example, reviewing programme targets and the 
means to reach those targets on a periodic basis can provide oppor- 
tunities to learn what is and is not working in a particular activity or 
programme. This is discussed further in Chapter 2. 


Often the information collected, the way it is collected and the 
way it is presented will vary considerably because of its intended 
use. Defining the purpose and scope of an evaluation is therefore an 
integral, preliminary step. Additionally, evaluations should distin- 
guish between outcomes that result from the services themselves 
and those that result from uncertainty or chance. Often this distinc- 
tion is blurred, but stating what is unknown is very important to the 
long-term viability of the management process. 


Minimizing conflict 

It is important that evaluations be used constructively. An evalua- 
tion should go beyond simply justifying past actions and identify- 
ing past inadequacies. The information should be directed towards 
future decisions and actions. Evaluations are not meant to be a judge- 
ment that affects staff on a personal level. Instead they should serve 
as an educational tool for both those evaluating and those being 
evaluated, and should focus primarily on institutions and policies 
rather than on the actions of individuals. Since evaluation involves 
judgement, the character and purpose of this judgement are impor- 
tant. As mentioned before, the primary reason for evaluating any 
health service is to learn about the implementation of policies, de- 
termine their consequences, and feed this knowledge and informa- 
tion into future efforts to formulate and implement policy. 


Undertaking a full-scale evaluation of services can be risky for 
an organization. A participatory approach from the outset is highly 
encouraged, particularly when the performance of personnel is part 
of an overall evaluation; otherwise, an adversarial relationship may 
evolve between evaluators and those evaluated. For example, per- 
sonnel being evaluated may perceive the evaluators as opponents 
who try to take away funding or reduce staffing, and evaluators may 
characterize personnel being evaluated as obstructive to the process. 
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Both these conflicting claims may be legitimate in certain situations. 
It is a manager’s task to recognize and prevent resistance to evalua- 
tion. Managers can emphasize that learning about implementation 
of policies, or determining the specific outcome of a service, does 
not necessarily mean taking funds away from existing programmes. 
In addition, if from the start the individuals being evaluated know, 
or help develop, the criteria on which they will be judged, the entire 
procedure is likely to be less intimidating and confrontational. Em- 
phasis on the educational aspect of an evaluation goes a long way 
towards reducing confrontation between those being evaluated and 
those evaluating. 


There is great potential for distorting the meaning of evaluation. 
Many health programmes and services claim to evaluate, yet they 
do not carry out any systematic analysis of their efforts. Although it 
may be difficult to evaluate a service in a strictly scientific way, the 
most systematic evaluation procedures available should be utilized. 
Similarly, avoiding simplistic conclusions from evaluations is also 
necessary. It is difficult, but tempting, to classify health programmes 
and services as either successful or unsuccessful, good or bad. Usu- 
ally, such designations are of little value because they omit further 
explanation and discussion of the factors contributing to programme 
success or failure. Furthermore, uncertainties and unanswered ques- 
tions about an evaluation will always exist. Though completely avoid- 
ing these unknowns is impossible, stating them explicitly is necessary. 


THE SUBJECT: WHAT CAN OR SHOULD BE EVALUATED? 


The subject of an evaluation should reflect its intended purpose. For 
example, if the main purpose of an evaluation is to determine whether 
the needs of the population are being met, then the subject will be 
the relationship between the specified goals and the identified needs 
in the population. All the different components of the service, either 
individually or in conjunction with other components, may be evalu- 
ated. Fig. 5 illustrates the composition of a generic health service. 
All governmental services and programmes begin at a certain point 
in time, then they alter inputs and activities to create outputs and 
desired outcomes. Inputs are the resources, including the raw mate- 
rial, money, personnel, training and time, that are used to plan and 
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Fig. 5. Generic health service model 
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carry out different activities. Process refers to the transformation of 
inputs into activities and outputs, which are the results of the planned 
activities. Outcome can be thought of as the change in the popula- 
tion as a result of the outputs. The food inspection service is a sim- 
ple example of an environmental health service. Personnel and money 
(inputs) are budgeted for a certain number of inspections (activities) 
during a given period of time. As a result of these inspections, some 
establishments are fined or closed because of health code viola- 
tions. These results are the outputs. The reduction in, or lack of, 
food poisoning episodes in the population are then said to be the 
outcome of the food inspection service. 


Confusion may arise about the distinction between output and 
outcome. The following example may help in better understanding 
these concepts. The Regional Office for Europe of the World Health 
Organization has specified 21 health targets for the region (4). Gen- 
erally, these targets correspond to outputs — that is, measurable, spe- 
cific results. Together, the targets contribute to the overall goal of 
the World Health Organization, ensuring health for all in the twenty- 
first century. This goal, like others (such as becoming a more sus- 
tainable society), is related to outcome — that is, more general, popu- 
lation-wide results that may be difficult to measure. Targets, there- 
fore, generally refer to measurable, specific actions or results. On 
the other hand, goals refer to more general sets of actions, which 
may be hard to measure. When planning an activity, it is useful to 
set targets that are measurable: this may facilitate the production of 
meaningful results when evaluating programmes or services. 


In the past, WHO has defined six basic subjects for an evalua- 
tion: relevance, adequacy, process, impact, efficiency and effective- 
ness (1). To these subjects we add equity and sustainability. Each 
subject is discussed in some detail in the following sections. Defini- 
tions for these terms are based on those found in the WHO publication 
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on guiding principles for health programme evaluation (1). The sub- 
jects are presented in a specific order intentionally. Ideally, relevance 
and adequacy should be the first items explored and evaluated. For 
example, the impact of a service is irrelevant if it does not respond 
to a specific need in the population. 


Relevance 

The ability of programmes, activities, services and institutions to 
respond appropriately to human needs (reflected in social and eco- 
nomic priorities) should be evaluated. Relevance relates these dif- 
ferent components of environmental health services to the social 
and economic policy goals of the community. In other words, rel- 
evance refers to the appropriateness of the service. For example, 
even if a service caused a change in health status, that change may 
not be related to the changes envisaged by the planners — that is, it is 
not relevant to their pre-established needs. This concept is illustrated 
in Fig. 6 as the relationship between the whole community and the 
health service, because human needs provide the basis for the rela- 
tionship. Goals and objectives should also be evaluated for relevance. 


Adequacy 

Adequacy refers to sufficient attention being paid to certain previ- 
ously determined courses of action, such as the various issues to be 
considered during broadly based programming. In other words, do 
the goals specified sufficiently address the needs identified? Both 
the goals themselves and the relationship between goals and ser- 
vices can be evaluated for adequacy. 


Fig. 6. Evaluation subjects: relevance and adequacy 
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Process 

Process refers to the relationship between inputs and outputs (see 
Fig. 7). Evaluating a process deals with comparing actual activities 
with those scheduled. The reasons for achievements and shortcom- 
ings should be noted, along with remedies for shortcomings. The 
purpose of a process review is often to facilitate the monitoring of 
day-to-day activities, the supervision and implementation of ongoing 
activities, milestones, personnel matters, supplies and equipment, 
and the monitoring of how money is spent in relation to the budgets 
allocated. Process evaluations also allow managers to discriminate 
between problems that arise from the planning of activities and those 
that arise from preparing for and implementing activities. 


Impact 

Impact is an expression of the overall effect of a service on health 
and related socioeconomic development. The evaluation of impact 
is an attempt to measure the more encompassing population-based 
consequences of the services. The outcome is thus compared to the 
initial situation. In general, it is much easier to study process than to 
study impact. 


Efficiency 

Efficiency refers to the direct relationship between the results ob- 
tained from a service (outputs) and the amount of effort expended 
in terms of human, financial and other resources (inputs), health 
processes, technology and time. Efficiency demonstrates how well 
the outputs have been produced from the inputs. The aim of assess- 
ing efficiency is to improve implementation, but it also serves to 
identify the strengths and weaknesses of monitoring. Financial 


Fig. 7. Evaluation subjects: process and impact 
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auditing is a particular type of monitoring that often aims to improve 
efficiency. Placing a monetary value on inputs, outputs and outcome 
is one method of establishing programme efficiency. This, however, 
is not an easy task, especially for measurements of outcome. In fact, 
economic quantification of environmental impacts and environmen- 
tal health impacts is the subject of much debate in the field of environ- 
mental economics (more about efficiency can be found in Chapter 4). 


Effectiveness 

Effectiveness expresses the degree of attainment of the predeter- 
mined objectives and targets of a service. Effectiveness is assessed 
in order to improve the formulation of services, objectives and goals. 
Qualitative analyses of effectiveness are often preferred to quantita- 
tive analyses because they provide an in-depth, specific insight into 
context and circumstances (more about quantitative and qualitative 
analyses can be found in Chapter 4). Assessment of effectiveness 
should also be carried out with the community in mind: their satis- 
faction and dissatisfaction with services are essential to a successful 
programme. When combined with adequacy, effectiveness repre- 
sents the degree of attainment by the services themselves. For exam- 
ple, a cursory evaluation may show that environmental health ser- 
vice targets are being met. More detailed research, however, may 
show that this success is not the result of the service’s activity, but is 
instead the result of a wholly different development, such as the 
status of the national economy. Thus, the impact of the service may 
not be particularly effective. An important distinction between ef- 
fectiveness and impact is that effectiveness focuses broadly on the 
contribution of a particular service in attaining the predetermined 
targets of a service or policy, while impact does not. 


Equity 

Equity implies “that everyone should have a fair opportunity to at- 
tain his or her full health potential and, more pragmatically, that no 
one would be disadvantaged from achieving this potential” (1). Equi- 
ty, however, is a particularly difficult subject to evaluate because it 
can be defined in a number of different ways, even for the same 
situation (3). Owing to differences in socioeconomic conditions, 
spending the same amount of money in different areas can have 
different results. For example, in providing for clean drinking-water, 
there are at least three ways to define equity: 
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1. in terms of input: authorities may decide to spend equal 
amounts of money on water supply in both richer and poorer 
areas; 


2. in terms of outputs: authorities may decide to spend money so 
that richer and poorer areas receive the same water supply 
services (more money may be needed in poorer areas to com- 
pensate for prior differences in services); and 


3. in terms of outcomes: authorities may decide to spend money 
so that richer and poorer areas have the same level of a certain 
disease (again, even more resources may be required in 
poorer areas to compensate for prior differences in services, 
as well as to compensate for differences in socioeconomic 
conditions). 


The definition chosen influences the results of the evaluation and, 
thus, should be adhered to throughout the process. 


Sustainability 

At the United Nations Conference on Environment and Develop- 
ment, held in Rio de Janeiro, Brazil, in 1992, sustainable develop- 
ment was defined as “development that meets the needs of the present 
without compromising the ability of future generations to meet their 
own needs” (10). World governments recognized that economic de- 
velopment, health status and environmental resources are closely 
related, and they agreed to pursue sustainable development. This is 
a long-term process and may not result in significant health changes 
in the short term. In addition, sustainable does not necessarily mean 
healthy, because promoting sustainable development may remove 
or diminish attention to important health issues (//). 


Programmes and procedures can be evaluated in terms of their 
sustainability. An intervention should be sustainable not only in terms 
of environmental resources, as suggested at the Rio conference, but 
should also be sustainable in terms of financing, personnel commit- 
ments and political will. Despite growing attention to sustainable 
development, the ability to conclude that a particular service is or is 
not sustainable remains elusive. So far, it has only been possible to 
argue that one activity is more or less sustainable than another. 
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In summary, the subject of an evaluation may vary widely, de- 
pending on the questions that managers wish to answer. In addition, 
the order in which the above subjects are addressed is important. For 
example, impact cannot be evaluated unless the evaluator is reason- 
ably sure that the programme has been implemented as planned (that 
is, unless a process evaluation has been done), and measuring effi- 
ciency cannot be done without first knowing the impact of a service. 


WHO EVALUATES AND WHEN? 


Choosing the appropriate evaluator is an important concern that is 
closely related to the designated purpose of any evaluation. Desig- 
nating a team of people from relevant sectors who have an interest 
in planning and managing the progress of the evaluation is an exer- 
cise frequently performed to determine the subject, scope and time 
frame of the evaluation. Evaluators should possess the necessary 
skills and knowledge to do an evaluation. One of the purposes of 
this publication is to help provide these skills. 


Internal and/or external 

It is important to consider the position of the evaluators and their 
relationship to the programme being evaluated (/2). Internal evalu- 
ators are affiliated with the organization being evaluated and exter- 
nal evaluators are not, and each has its strengths and weaknesses. 


Internal evaluations are useful because the analysis is likely to 
contribute directly to the information needs of an organization. 
Moreover, data collection is more straightforward and less expen- 
sive. The credibility of internal evaluations, however, is often chal- 
lenged because the results may only have limited use outside the 
organization, the potential subjects for evaluation are narrow and 
the evaluators are perceived as being biased. 


External evaluations are more effective when addressing broad 
governmental issues and programmes between different sectors of 
government. Results may be more widely used because external 
evaluators have greater access to decision-making processes and 
are perceived as having more authority and credibility. In compari- 
son with internal evaluators, professional evaluators (consulting firms, 
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auditing offices, academic institutions) may be more experienced, or 
possess a broader range of evaluation skills. Lack of familiarity with 
the programmes to be evaluated may, however, limit external evalua- 
tions, and managers may be less willing to accept an outsider’s view. 


Often, especially in larger evaluations, both internal and external 
reviewers are employed, which helps to prevent conflicts that arise 
from the perceptions that outside evaluators do not fully appreciate 
a programme and that internal evaluators are not objective. The best 
combination of internal and external evaluators will depend on the 
programme or service to be evaluated. 


Timing and frequency 

The frequency of evaluations is an important decision. It should be 
often enough to produce meaningful information, but not so often 
as to prevent the progress of the work at hand. It is natural for smaller- 
scale evaluations to occur more frequently than large-scale evalua- 
tions. For example, it is not feasible to conduct large-scale evalua- 
tions on a weekly or even a monthly basis. But it is also undesirable 
for them to occur only at the end of a project or at an intervention. 
The monitoring and information systems should be able to provide 
information on a regular basis, thus allowing infrequent full-scale 
evaluations. Midterm evaluations may be a viable solution to the 
issue of frequency. The optimum time frame should be derived from 
a consensus among advocates and designers of an evaluation. It is 
important to remember that evaluation should reflect the continuing 
nature of the management process. Sometimes, evaluations are seen 
as discrete events that occur only at the end of a project, service or 
intervention. Although it is recommended that projects always be 
evaluated at their conclusion, this is clearly not the only time for an 
evaluation; less formal information and monitoring systems should 
identify problems as they arise throughout a programme’s imple- 
mentation. The essential task is to pass on correct information in 
time to make the necessary decisions. Thus, the evaluation manager 
needs to strike a balance among such competing factors as time, 
cost and personnel constraints in order to determine the optimal time 
frame and optimal format (formal or informal) for a major evalua- 
tion of the information collected. 


Box 2 summarizes the key recommendations of this chapter. 
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Box 2. Key recommendations from Chapter 1 





Recognize evaluation as a part of the ongoing cyclical management process 


Identify interested and affected parties as soon as possible; develop political 
and community support by bringing them into the decision-making process 


Keep evaluations open to new circumstances, new facts, new opinions, new 
ideas and new stakeholders 


Develop and follow a written plan for evaluation; disseminate it to the stakeholders 
at the beginning of the project (see Box 1) 


Prevent the perception that evaluation is a judgemental activity; stress its edu- 
cational and constructive nature 


Narrow the scope of the evaluation according to its intended purpose 


Strive to develop a clear monitoring and information protocol that delivers im- 
portant information to decision-makers but limits unimportant information 


Include a mixture of internal accounting of programme operations and external 
reviews of them 


Distinguish between outcomes that result from the services themselves and 
those that result from uncertainty or chance 


Avoid oversimplified conclusions 


Be both flexible and action-oriented in order to guide future planning and 
decision-making 


Promote issues of equity, intersectoral collaboration and community partici- 
pation 
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Theoretical Framework 
for Evaluating 
Environmental Health Services 


The previous chapter presented general principles and con- 
cepts of evaluation. This chapter builds on that foundation 
and applies the same concepts specifically to environmental 
health services. It begins by defining terms and briefly track- 
ing the historical development of environmental health ser- 
vices. Next, reasons for evaluating environmental health 
services are provided. Then, some of the particular difficulties 
of environmental health service evaluation are discussed. 
The last two sections explore the evaluation subject, authors 
and audience. 


CONCEPTS AND DEVELOPMENT OF ENVIRONMENTAL HEALTH 
SERVICES 


Environmental health is a relatively new term that has been used 
to define both a condition affecting human health and a profes- 
sional discipline. The term has evolved and has come to mean 
different things to different people. Recent international meet- 
ings (1—3) have attempted to develop consensus definitions of 
environmental health and environmental health services. The terms 
described here have been generated and accepted at these inter- 
national meetings and have been used in previous volumes in 
this series (4-7). 
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Environmental health 

Many countries have difficulties in working with the definitions of 
public health, sanitation, hygiene and environmental health, and of- 
ten freely interchange the terms. Questions also arise about the rela- 
tionship between healthy lifestyles and environmental health. These 
difficulties are further magnified when terms are translated into other 
languages, which naturally reflect different cultural and historical 
developments. In many countries, the term environmental health 
arouses interest because, historically, environmental and health mat- 
ters have been dealt with in a generally unrelated manner. The WHO 
Regional Office for Europe has been actively pursuing and promot- 
ing a common definition for environmental health (4): 


Environmental health comprises those aspects of human health, including 
quality of life, that are determined by physical, chemical, biological, social 
and psychosocial factors in the environment. It also refers to the theory and 
practice of assessing, correcting, controlling and preventing those factors 
in the environment that can potentially affect adversely the health of present 
and future generations. 


Environmental health comprises diverse interest areas, including 
but not limited to: air pollution, food safety, the built environment, 
injury prevention, land use planning, noise pollution, occupational 
health, chemical safety, radiation, tourism, waste management and 
water management. As with other disciplines, including infectious 
disease control, health education, epidemiology, health research and 
mental health, the discipline of environmental health serves as one 
component of public health. 


One of the main goals of integrating environment and health is to 
look at many different sectors with a holistic view. The exact mix- 
ture of sectors to be included in the discipline of environmental health 
will inevitably be defined differently by different countries, and 
debating the inclusion or exclusion of particular sectors would not 
serve the purpose of this publication. The point to be emphasized is 
the need to measure and explore more broadly the interactions and 
interrelationships of the different sectors. 


Environmental health services 
The term environmental health services is also a relatively new con- 
cept and is defined as follows (5): 
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Environmental health services can be defined as those services implement- 
ing environmental health policies through monitoring and control activi- 
ties. They carry out that role by promoting the improvement of environ- 
mental parameters and by encouraging the use of environmentally friendly 
and healthy technologies and behaviour. They also have a leading role in 
developing and suggesting new areas of policy. 


This definition implies a very specific meaning for the word ser- 
vice — that is, it refers to the institutions that implement, create or 
otherwise address environmental health issues. But the term envi- 
ronmental health services is used equally to describe the work and 
activities that are carried out by these environmental health institu- 
tions and agencies. Environmental health services act as the direct 
interface between the policy-makers and the subjects of that policy. 
They also relate directly to the general public, in dealing with their 
complaints and concerns about environmental health issues. There 
is a need for such services to be appropriately targeted and sympa- 
thetically responsive to public needs, while at the same time being 
representative of the views of the authorities (local, regional or na- 
tional government). One of the more difficult tasks for any service 
delivery is to ensure that services are appropriately targeted and re- 
sponsive, and this is particularly relevant for environmental health 
services. Environmental health service managers require accurate 
assessments upon which decisions for the future can be based; how- 
ever, the methodology for performing such assessments in the envi- 
ronmental health arena is not very strong. 


An improved methodology to gauge the process, impact, relevance 
and adequacy of services is required. Such a methodology would 
help to raise the quality and quantity of information reaching man- 
agers, thereby strengthening their ability to make decisions about 
the delivery of future services. In addition, methods for evaluating 
health services can also help in developing more clearly the role of 
a service and its formal strategy. This volume aims to fill some of 
these gaps. 


Historical development of environmental health 

The issues addressed by environmental health services must be de- 
termined for a specific situation and are particularly dependent on 
socioeconomic circumstances and traditions. It is impossible to ar- 
gue that present expenditures for preventive health measures will 
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abolish the need for future expenditures. Clearly, the nature of hu- 
man ailments has changed in the past and will continue to change in 
the future, constantly requiring new resources. It is the job of public 
health professionals to track that change as it occurs and to try to 
predict future changes in order to provide the best health services 
for the population. The diagram shown as Fig. 8 has been used to 
model the long-term development of two aspects of public health: 
changes in disease patterns and subsequent professional responses. 


Health transition theory suggests that in most countries the char- 
acter of disease has changed with industrial development. Sanitary 
diseases (that is, acute health responses associated with poor living 
conditions, improper sanitation and overcrowding) have given way 
in many areas to “environment and lifestyle” diseases, which gener- 
ally develop over a longer period of time and have less direct links 
to the environment. Similarly, as these diseases are better controlled, 
psychosocial diseases, such as mental depression, will need to be 
addressed. The focus of professional development is changing from 
a substantially medical and engineering discipline to social, behav- 
ioural and management disciplines. 


The model appears applicable to most countries in the European 
Region, although certainly each country might fall on a different 
place along the time axis. The location depends primarily on the 


Fig. 8. Professional responses to changing disease patterns 
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level of economic wealth within a country and its individuals. Simi- 
larly, different regions or localities within a country (and even within 
cities) might appear in substantially different places along the dis- 
ease axis. This can be clearly demonstrated by the rise of tuberculo- 
sis and other “diseases of the poor” in many declining areas of large 
cities. It is possible to move along the time axis, but as progress 
occurs the requirements of the environmental health services will 
change. Once the basic needs for survival of a population are ad- 
dressed, the possibilities for environmental health activities seem only 
to be limited by resources. Nevertheless, the problem of scarce re- 
sources is a serious one throughout the Region; thus, in all coun- 
tries, environmental health services should be carefully chosen and 
planned to have a specific impact. Again, this publication does not 
aim to prescribe which environmental health goals national, regional 
or local authorities should choose; they have to complete this task 
on their own. It aims instead to provide a methodology for evaluat- 
ing the services provided at a given time, and after the decisions 
have been made. Once the tools for evaluation are used, and a well 
organized evaluation plan is established, prioritizing services should 
become easier. 


Some of the disciplines addressed here will not be applicable 
immediately to all countries in the Region, but will take on a grow- 
ing importance as their development progresses according to this 
model. Applying the theory in this way will give countries along the 
first part of the time axis opportunities to share experiences with 
others, thereby possibly moving them further along the time axis. 
Integrated information sharing may allow countries to better prioritize 
services and to utilize or apply research completed elsewhere. This 
volume is especially targeted at those countries currently in the mid- 
dle of this process — that is, countries with enough economic strength 
or sufficient political will to devote substantial resources to the pre- 
vention of diseases of environment and lifestyle. 


Fig. 8 also serves to illustrate the changing qualities needed by 
health-related professionals to address problems associated with 
changes in disease patterns. How the progression and rise of one 
type of disease occurs is open to speculation. Perhaps as one disease 
begins to be cured the others become more apparent, or perhaps the 
public begins to demand progressive improvements once one disease 


38 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5 


has become manageable. The progression and rise of diseases is 
also closely connected to broader developments, such as those in 
economy, culture and science. 


For diseases of environment and lifestyle, the skills required have 
changed from purely medical to those more focused on engineering 
and management. For example, it is already apparent that reports of 
diseases related to psychosocial factors are increasing and, in the 
future, will need to be addressed by professionals with different skills. 
This emphasizes the need for building competency within environ- 
mental health services. Two previous volumes in this series address 
these aspects of training and curricula development (6, 7). 


In general, the diagram depicted in Fig. 8 is relevant for most 
European countries. Where sanitary or hygienic problems have been 
predominant, the response has been to utilize the skills of doctors 
and engineers. This is the present situation in many of the countries 
of central and eastern Europe (CCEE), although the need for envi- 
ronmental health professionals trained in management and social 
science is now recognized. Facing the new environmental health chal- 
lenges is possible only if service staff have the appropriate and var- 
ied skills necessary. The breakdown of traditional roles does, how- 
ever, take time and a lag phase is inevitable. It is very difficult to 
predict and prepare for future service needs, but this is a basic func- 
tion of all managers and politicians. Evaluation of current services 
is the foundation upon which decisions about the future are based. 


WHY EVALUATE ENVIRONMENTAL HEALTH SERVICES? 


There are numerous reasons for carrying out an evaluation of an 
environmental health service. Primarily, it is important to address 
the general management goals of evaluating the effectiveness, effi- 
ciency, impact, process, relevance and adequacy of policies, activi- 
ties, objectives, services and goals. Evaluations can be used to de- 
termine what has been achieved by the service, how the service 
works, and how to improve the efficiency and process of services. 
Without evaluating environmental health projects, it is difficult to 
ascertain whether and how well objectives have been achieved. Thus, 
an examination of the capabilities and effectiveness of services can 
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be used to assess whether policies have been effective in achieving 
their objectives. In addition, evaluations provide valuable informa- 
tion about the process of delivering services. Furthermore, learning 
more about how the services operate may demonstrate problems in 
the implementation process. Knowing what the problems have been 
helps define ways to improve the efficiency and effectiveness of the 
service. 


As a tool, improved evaluation also assists managers in choosing 
between the various types of environmental health service that an 
authority can provide and, perhaps, in prioritizing them. In the past, 
the mixture of environmental health services that authorities have 
chosen to provide has rarely been the result of scientifically calcu- 
lated or strategic planning. A careful evaluation of the benefits of 
current services could identify unnecessary redundancies, or gaps 
that require filling. Unfortunately, while it is relatively easy to find 
material that provides advice on how to design and implement spe- 
cific environmental health services (for example, waste disposal or 
food inspection), there is little guidance on how to choose between 
different services, or between specific actions. One can ask: Given 
limited resources, which service has the better impact on the health 
of the population? One can also ask: Which has a smaller, negative 
impact on the environment? Both questions are important, but not 
easily answered, given the current methodology and practices of 
service delivery. 


Evaluations also serve to improve the knowledge of programme 
impact. This knowledge in turn helps justify expenditures on 


Box. 3. Primary reasons for evaluating 
environmental health services 





Draw upon experiences to improve overall service delivery 

Improve efficiency and process of services 

Determine effectiveness and impact of goals, objectives and services 
Ascertain relevance and adequacy of goals, objectives and services 
Describe expenditures on environmental health services to political leaders 
and communities 

Recognize and respond to public needs and wants 

Prioritize research and plan for the future 

Prioritize actions and decisions 

Find allies in other agencies, services or sectors 
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environmental health services to political leaders and communities. 
An evaluation of services may also act as a mechanism through which 
changes in public needs and wants can be recognized. In order to 
evaluate any of the service goals, objectives, outputs and outcomes, 
managers first need to know public expectations and wishes. A 
proper method of evaluation helps to identify inadequate and irrel- 
evant services and helps to ensure that such services will be ad- 
justed accordingly. 


Additionally, an evaluation may be used as a mechanism for 
prioritizing research. “Further research required” is a familiar phrase. 
Often, public and environmental health officials do not have the 
luxury of waiting for additional research; instead, they make deci- 
sions based on the best available information. In such cases, the 
precautionary principle” is applied and the need for more research 
remains. It is therefore important that the most relevant questions be 
addressed first, and evaluation is a tool for determining where to 
begin. Research is also tied to the need for applying the knowledge 
gained from an evaluation to making decisions about the future, 
because once the gaps have been defined, research can be directed 
to address these gaps. A balance should be struck between the need 
to act immediately and the need to wait for further information. 


Evaluation may also foster interactions between agencies and or- 
ganizations that ordinarily do not cross paths. Through these inter- 
actions, personal relationships and alliances between individuals with 
similar objectives and perspectives may be developed. 


As stated in Chapter 1, managers should ask themselves why 
they are evaluating services (Box 3). Sometimes an evaluation is the 
result of economic or financial considerations; in other cases man- 
agers may simply wish to know how much progress has been made. 
The need to inform and convince the public about the value of the 
service is also an important underlying reason to evaluate. Evalua- 
tion may, in some cases, open the door to two-way communication 
with the public (see Chapter 5 for more information on communication 


> The precautionary principle instructs managers to take action in the interest of 
protecting public health, even when fully conclusive evidence is unavailable. In other 
words, when faced with large uncertainties, managers are encouraged to err on the side of 
overprotection. 
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issues). Finally, the value of having an integrated programme of serv- 
ices is greater than the simple sum of its components. A thorough 
evaluation of programmes may provide examples of this to service 
managers. 


BARRIERS TO EVALUATING ENVIRONMENTAL HEALTH SERVICES 


Environmental health services are particularly difficult to evaluate 
through traditional mechanisms. Many years ago a WHO working 
group (8) identified several of these difficulties, but most of them 
have not yet been adequately addressed. There are six main barriers 
to effective evaluation. 


1. Environmental health services try to produce a non-event. How 
can it be proved that a non-event is the result of a service? 


2. Services are administered by several agencies, such as health, 
environment, agriculture, industry and transport. 


3. Environmental health services address three different time pe- 
riods: the past, present and future. 


4. There are many weak and indirect links between environment 
and health effects; long latencies are usually associated with 
known environmentally caused diseases. 


5. It is difficult to define the community and, thus, to determine 
the relevance and adequacy of the services. 


6. Unreliable information limits long-term comparisons. 


An important way to overcome these barriers is to recognize and 
discuss them with the evaluation team at the beginning of and 
throughout the process. The remainder of this section discusses each 
of these barriers and suggests ways in which they can be addressed. 


Proving a negative 
Many environmental health services strive to produce non-events; 
in other words, the purpose of many environmental health services 
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is to prevent an adverse effect from occurring. Thus, if the services 
are well targeted and managed properly, their effects will be to pro- 
duce the absence of an event. If there are no events to measure or 
discuss, how can a manager prove that the services are working prop- 
erly? Similarly, it is a problem that many environmental health ser- 
vices work to maintain the status quo. Measurement is especially 
difficult when health effects fall below what is detectable individu- 
ally, or what is unacceptable socially. So, while it may be easy to 
identify services that are not working properly (complaints from citi- 
zens generally provide a good warning system), it may be difficult 
to prove that services are working properly. Typically, these prob- 
lems have been overcome by identifying changes in health status, in 
environmental parameters or in indicators of them that, given a proper 
design, might be related to service activities. 


Multi-agency responsibility for environmental health 
services 

Environmental health services have traditionally been administered 
through more than one governmental agency, thus making coordi- 
nation of evaluation efforts difficult. Typically, it is not just minis- 
tries of health or of environmental protection, or both, that deliver 
environmental health services. Many other sectors and their minis- 
terial authorities, such as agriculture, industry, energy, transport, tour- 
ism and labour, are involved. Generally, none of these authorities 
knows what services the other sectors and authorities are providing, 
and so a comprehensive evaluation of the effects of the services is 
not possible. There have been recent efforts to promote inter- 
sectorality. The Helsinki Declaration (9), for example, places par- 
ticular emphasis on improving coordination among different gov- 
ernmental authorities dealing with subjects that relate to environ- 
mental health. Additional efforts, however, are required to increase 
the probability of a sufficient evaluation; for instance, a coordinated 
evaluation of all environmental health services in the relevant sec- 
tors could be incorporated into NEHAPs. 


Past, present or future? 

Another barrier to effective evaluation methods is that environmen- 
tal health services must address and act in three different time peri- 
ods: the past, present and future. These services must repair past 
damage, protect the present environment and provide good practices 
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in the future. Policies cannot simply address only one of these peri- 
ods of time; a balance should be made among them. For instance, it 
may be difficult to quantify the value a particular resource will have 
in the future. 


Inadequate links to health outcomes 

Despite the advances of the past 20 years in understanding the links 
between exposure to environmental agents and its impact on health, 
there are still many questions and unproved theories. This lack of a 
clear scientific link has hindered the development of comprehen- 
sive indicators that measure such health effects. Furthermore, the 
long latency periods associated with many environmentally caused 
health effects create difficulties for comprehensive evaluations. Many 
types of environmental health service are trying to prevent health 
effects that occur over a long period of time, and that only rarely 
manifest themselves in the general population. Programmes that are 
implemented today may not produce measurable impacts on health 
for months or sometimes years. Thus, the use of traditional epide- 
miological methods to measure and evaluate the impact of pro- 
grammes is often inadequate. 


The entire community is the client 

Evaluating environmental health service activity can be done within 
the broader context of its clients or target population. Generally, it is 
easier to evaluate a service directed at a well defined, limited popu- 
lation; however, unlike some health services that have more or less 
defined populations (for example, mothers and children of low so- 
cioeconomic status in many maternal and child health services), the 
community as a whole is usually the “client” of environmental health 
services. In many cases, it is therefore very difficult to narrow or 
further define and measure the “community” that benefits from par- 
ticular services. In such cases it may also be problematic to promote 
or evaluate equity — that is, it is hard to identify which individuals 
receive attention, protection or care, and which do not. When the 
entire community is the client, it is sometimes difficult to strengthen 
a manager’s abilities to improve services that are based on the needs 
and wants of the population. This, however, is not always a major 
problem. For example, environmental health services may be focus- 
ing only on specific risk groups, such as children or the elderly, that 
are in specific need of help. This focus on a well defined, limited 
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population facilitates population-based measurements in evaluation 
research. 


Data of unknown quality 

One of the biggest barriers to effective evaluation is unreliable in- 
formation. Because of their nature, most environmentally caused 
health effects require analyses of long-term health trends. Though 
there is a great deal of long-term health and environmental quality 
monitoring data available for the CCEE and the newly independent 
states (NIS), the validity and reliability of much of these data have 
been questioned (for example, see the Estonian case study in Chap- 
ter 6). A decade after the fall of the Berlin wall, countries, and some- 
times local authorities, are developing their own monitoring sys- 
tems that are occasionally incompatible with each other. This makes 
comparisons between and within countries unreliable. 


Problems of data quality and compatibility are not limited to the 
CCEE and may even cause difficulties in western Europe. For exam- 
ple, significant advances in technology in the past few decades have 
improved water sampling techniques, making them much more sen- 
sitive to toxic substances. In some areas, current raw data, when 
compared with data from as recent as 5 years ago, appear to indicate 
that drinking-water is becoming more toxic. It is much more likely, 
however, that the change is due to more accurate sampling tech- 
niques than to truly higher levels of toxicity. But it is still a problem 
for managers to distinguish between these two scenarios, especially 
if they are trying to use the level of toxicity in drinking-water as an 
indicator of success of their water protection service. It is a dilemma: 
on the one hand, it is desirable to utilize new technologies as they 
become acceptable and unrealistic to expect managers to ignore such 
technologies; on the other hand, managers clearly have to act to 
ensure that their water sampling techniques are consistent and that 
they produce accurate and reliable results. 


Tools from other disciplines 

The barriers to evaluating environmental health services have not 
been adequately addressed or resolved since a WHO working group 
(S) met almost 30 years ago. Furthermore, barriers such as poor data 
quality are unlikely to be fully resolved in the near future. Admit- 
tedly, there is much to be done in certain areas, but this should not 
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be used as an excuse for not conducting evaluations. Evaluation is 
absolutely necessary for the delivery of quality environmental health 
services. Much can be done immediately to improve evaluations 
based on tools used by other disciplines. The remainder of this book 
describes some of these tools in some detail and suggests how they 
might be adapted and applied in the context of environmental health 
Services. 


WHAT ASPECTS OF THE SERVICES ARE TO BE EVALUATED? 


As stated in Chapter 1, what is evaluated (the subject) is closely 
related to the goal or purpose of the evaluation. It is strongly recom- 
mended that a team of interested parties first clearly define the pur- 
pose of an evaluation and then decide the subject and scope of the 
evaluation. A balance should be reached between extremely spe- 
cific evaluations, which provide more detailed information on a lim- 
ited range of topics, and general evaluations, which provide conclu- 
sions that are less specific but may be more widely applicable. 


It is impossible for authorities to evaluate environmental health 
services in their entirety at one time. The tasks and duties of envi- 
ronmental health services are simply too vast and diverse. Further- 
more, resources are not likely to cover evaluation of each subject 
(such as efficiency or relevance) for each of the different aspects of 
environmental health services. In this context, “aspects” refers to 
the various programmes that can be evaluated (such as accidents, 
waste management and laboratory practices) or to management ac- 
tivities (such as enforcement, financial management and training). 
Individual countries and districts have to prioritize evaluations ac- 
cording to local circumstances. Fig. 9 presents three questions re- 
lated to the subject of an environmental health service. This figure is 
intended to serve as a guide and to initiate discussion, but is not 
intended to relate all potential services that fall within the realm of 
environmental health. 


It is clear that both specific programmes (interventions) and dif- 
ferent elements of management should be evaluated, so these two 
areas have been separated in Fig. 9. As the figure suggests, all types 
of management and intervention can be evaluated, yet some will 
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Fig. 9. The subject of environmental health service evaluations 
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l (2) What part of service delivery should be evaluated? 


( Input ) ( Output ) | Outcome ) l Process ) | Impact ) 


(3) What subject or criteria should be evaluated? 


Effectiveness J Relevance N Adequacy ) Equity ) Sustainability ) 


inevitably be more useful than others. Some examples have been 
singled out in Box 4, with the intention of clarifying what each type 
of evaluation achieves for the evaluator. Examples of questions that 
might be answered by each type of analysis are included in the brief 
descriptions that follow. In addition, these examples are meant to 
provide a more concrete understanding of a manageable subject for 
an evaluation. 















( Efficiency 


WHo AND FOR WHOM? 


As stated in Chapter 1, it is recommended that a team of interested 
people be selected to manage large-scale evaluations. Creating a team 
that includes a mixture of people directly involved with the programme 
(or decisions to be evaluated) and impartial reviewers with evalua- 
tion experience is often the preferred strategy. Depending on the 
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Box 4. Examples of what to evaluate 





Inputs and outputs of training personnel 

Analysing inputs and outputs determines the utilization of resources and the 
delivery of planned activities. For training personnel, this might include an analysis 
of the cost, supplies, personnel and person-hours (the inputs) and an analysis 
of the number of people trained and the number of training sessions completed 
(the outputs). Such an analysis might include the information provided in the 
training courses, but it would not give any indication of the quality of the ser- 
vices. 


Outcome of a car accident prevention programme 

Outcome is studied in order to measure changes that occur at the population 
level as a result of programme activities (outputs). For example, investigating 
the outcome of a car accident prevention programme would mean looking for 
changes in the number and types of car accident. If alcohol were an important 
consideration, the number of accidents related to alcohol might be measured. 
Also, a change in behaviour (such as the practice of wearing a seat belt) may 
be a key consideration. 


Relevance of a particular regulation 

Relevance pertains to whether the right thing is being addressed. Regulations, 
for example, can be studied for relevance. To do this, one might ask whether a 
risk being addressed is important enough to warrant regulation. For the case of 
anti-smoking regulations, one might want to determine whether they would be 
warranted in public restaurants. 


Adequacy of an air pollution monitoring service 

Adequacy attempts to measure whether enough of the right thing is being done. 
Air pollution monitoring services may be studied to determine whether the right 
substances are being monitored in the right place, at the right time, with the right 
techniques and for a long enough period of time. 


Process of waste management 

Process is the means by which inputs are transformed into outputs — that is, the 
activities that occur. Analysing the process of waste management services can 
identify important strengths and weaknesses in the procedure. By tracking the 
process of waste generation, collection and end-use/disposal, one can gain 
valuable insights into ways to improve processes and create incentives for 
recycling, reducing and reusing waste. 


Impact of food safety programmes 

Impact analysis is concerned with the broad, population-based consequences 
of services (outcomes). For example, fluctuations in food-related morbidity and 
mortality might be a measure of the impact of food safety programmes. 


Efficiency of the enforcement procedure 

Efficiency relates the results of a service (outputs) to the resources and efforts 
put into that service (inputs). It provides a gauge of how well plans are carried 
out. A monetary value is usually involved with assessments of efficiency. For 
the case of enforcement, a manager might look at the efforts to improve com- 
pliance to certain measures. What level of compliance is achieved, given the 
amount of resources devoted to the enforcement procedure? (In this case it is 
especially important to distinguish between voluntary compliance and that which 
is a result of active enforcement policies.) 
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Box 4 (contd) 





Effectiveness of staffing profiles 

Effectiveness relates outputs to the goals specified. In terms of staffing, a key 
question would be: Is there sufficient manpower available (with the necessary 
skills) to address the problems the service is facing? Can staff handle prob- 
lems effectively? 


Equity of built environment policies 

Equity is concerned with equal opportunities to fulfil health potential. One could 
ask whether a given policy is causing or perpetuating disadvantages in certain 
groups. For example, from the built environment sector, some urban redevelop- 
ment schemes disproportionately burden low-income groups by displacing low- 
income housing with upper- or middle-income housing. 


Sustainability of financial arrangements 

Sustainability generally refers to three types of resource: environmental, finan- 
cial and personnel. It is important to ask whether these different types of re- 
source are conducive to the long-term continuation of services, without deplet- 
ing the resources for future generations. For example, financial structures can 
be evaluated for sustainability by asking: Do the current financial arrangements 
provide for continuation of services? 








scope of the evaluation, the mixture of skills, training and experi- 
ence within the evaluation team should be diverse — the more com- 
plex the evaluation, the more diverse the mixture should be. The 
mixture would of course include expertise in the appropriate disci- 
pline and familiarity with the programme or process in question, 
and might include social scientists, statisticians and information 
managers. Having a management representative involved (even as a 
silent participant) can also help produce a useful evaluation. Some- 
times cross-disciplinary evaluations (drawing on professionals from 
other sectors) are warranted. Some programme directors have found 
that personnel from other sectors can bring a fresh perspective and 
can provide training and expertise to the evaluator. A range of ex- 
pertise in different disciplines is desirable to avoid the criticism of 
having a narrow perspective. An important goal is to perform a cred- 
ible evaluation by involving expertise appropriate to the scale of the 
task to be performed. 


Earlier in this chapter it is noted that environmental health ser- 
vices often act as an interface. Many different organizations interact 
and cooperate with environmental health services, including parlia- 
ment, academic and scientific sectors, the media, the general public 
and industry (see Fig. 10). It is important to remember this context, 
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Fig. 10. Environmental health services and 
cooperating organizations 























Parliament/Political parties National administration 
General public Media 
Nongovernmental organizations Environmental Environmental services 
health 
Science services Local administration 











Economy, trade, industry ii i Health service managers 








as evaluations may have different target audiences in different or- 
ganizations. 


Evaluations benefit the authorities who implement the services, 
those who pay for the services, those who plan for the future, those 
who direct research, the professionals who work with the pro- 
grammes, and the population at large. These are the groups for whom 
evaluations are conducted. Presumably any effect that a service has 
in a particular situation will be improved once the service is evalu- 
ated and adjusted according to the results. Once the evaluation is 
completed, the service will improve by becoming more responsive 
to public needs and wants, and by becoming more effective and 
efficient. This should improve the health of the population (which 
may or may not be measurable). In addition, a high return from a 
well targeted evaluation can be expected. It is important, therefore, 
to identify the target audience for which the evaluation is being per- 
formed. If the target audience is well specified, the task becomes 
clearer and it is more likely that the results of the evaluation will be 
useful to the service. 


All the examples in Box 4 have been based on the assumption 
that these evaluations are occurring “for managers”, that is, that man- 
agers are evaluating services for their own use and to augment their 
ability to improve the services. Naturally, there are additional groups 
“for whom” evaluations can be done. Thus, an additional dimen- 
sion can be added to Fig. 9. For each potential evaluation, a differ- 
ent group may have a different reason for evaluating the service. For 
example, when services are subsidized by donor agencies, these 
agencies may want to evaluate the process in order to illustrate 
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accountability; or they may evaluate sustainability in terms of per- 
sonnel requirements. Different types of population could also be 
considered; then, each such subdivision might include international 
groups, national groups, middle-sized geographical areas (for ex- 
ample, a county or district), local areas (for example, a town or city), 
or even a specific subpopulation based on exposure (for example, 
children, the elderly or pesticide applicators). 


Evaluating a service, a programme, or the management is an over- 
simplified notion. It is essential to clearly define and narrow the scope 
of the evaluation, particularly when faced with limited resources. Only 
specific aspects of the service, programme or management can be evalu- 
ated. The scope reflects the intended purpose of the evaluation, and 
thus may be narrowed by the type of service or programme to be evalu- 
ated, by any of the traditional evaluation subjects, by the intended target 
audience, or by the population affected. 


Box 5 summarizes the key recommendations of this chapter. 
Box 5. Key recommendations from Chapter 2 

Clearly define and narrow the scope of the evaluation so that it becomes both a 

meaningful and feasible activity 


Staff the evaluation team with individuals with the appropriate range of expertise 


Be aware of the specific features of the environmental health service that may 
affect the evaluation techniques and procedures 


Find ways to address or account for the barriers to evaluating environmental 
health services 


Consider the context surrounding various organizations that cooperate and put 
the delivery of environmental health services into that context 
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Data and Indicators 


Once the purpose, subject, scope, frequency and timing 
of an evaluation have been established, the evaluator 
must determine what to measure and how to measure it. 
The tools that make measurements possible are data and 
indicators; they are also the building blocks of comparative 
evaluation. This chapter concentrates on indicators, 
describing what they are, who uses them and why, 

and giving examples of five types of indicator that often 
contribute to the evaluation of environmental health 
services. 


DATA 


Data can be defined as individual points of observation that are 
collected on a particular subject. Collecting data on environmen- 
tal health services is not an evaluation goal, but is instead a means 
to gain an understanding of these services. Although critical to 
evaluation, data are only part of the evaluation process. By them- 
selves, data are a useless mass of figures (for example, on health 
status); but when they are put into a proper context and inter- 
preted, they become information that is useful to managers and 
decision-makers. 
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INDICATORS: DEFINITION AND PURPOSE 


Indicators measure change. They are variables and are used as sur- 
rogates for measuring a parameter of interest, when practical diffi- 
culties prevent the exact measurement of that parameter. In a sense, 
an indicator is an empirical model that accurately reflects an event, 
even though it is only an indirect or partial measure of a complex 
situation. For example, the presence of high concentrations of sulfur 
dioxide in air alone is not necessarily a health risk, but sulfur diox- 
ide concentrations are often used as an indicator of overall air 
quality. 


Indicators are particularly useful when measured over a period 
of time, so that they track direction, magnitude and rate of change. 
They are also useful for comparing different areas (or groups of 
people) at the same moment in time. Though indicators are usually 
indirect measures, they can assist managers in making inferences 
and projections that can facilitate decision-making and other man- 
agement functions. 


Three major applications for indicators can be identified: 
decision-making, communication and policy follow-up. 


Indicators support decision-making and comparisons by provid- 
ing information about efficiency, effectiveness and problem situa- 
tions. Decisions about priority of services, sequence of activities 
and allocation of resources are often based on a series of indica- 
tors. 


Indicators are critical for productive communication because they 
promote a common understanding through the use of a consistent 
framework for measurement. They allow international organizations 
and national and local authorities to communicate with each other 
and with their many partners (or stakeholders), including communi- 
ties, institutes and NGOs. 


Indicators also help determine the degree to which goals are met 
and policies followed. In particular, performance indicators are be- 
ing used increasingly to monitor implementation. For example, the 
Protocol on Water and Health, signed by the Member States of the 
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European Region at the Third Ministerial Conference on Environ- 
ment and Health in London, United Kingdom, in 1999 (1) mandates 
that the signing parties will: 


Collect and evaluate data on: 


(a) their progress towards the achievement of the targets [set forth in the 
protocol]; and 


(b) indicators that are designed to show how far that progress has contrib- 
uted towards preventing, controlling or reducing water-related disease. 


OBJECTIVES FOR INDICATORS 


This section presents a set of qualities (or objectives) that evaluators 
should seek for indicators (2). Only some of these qualities can be 
maximized simultaneously, because no single indicator can posses 
them all. A suite of indicators is commonly used to avoid the limita- 
tions of any one indicator. 


Validity 

An indicator should accurately measure what it is supposed to meas- 
ure. If the data from which the indicator is calculated are inaccurate 
or are subject to unforeseen variations, the indicator will not be valid. 
For instance, the validity of past data from the CCEE has been ques- 
tioned because the Soviet system fostered inaccurate record keep- 
ing. An example of this is discussed in Chapter 6 in a case study 
entitled Assessment of environmental monitoring in Estonia 
(page 145). Compliance with international standards (such as those 
recommended by the International Organization for Standardization 
(ISO)) helps secure data validity. 


Internal reliability 

When an indicator is reliable, it means that under similar conditions 
it allows the same inferences to be made by different people. A dis- 
tinction can be made between internal and external reliability. Inter- 
nal reliability means that an indicator should provide the same basis 
for conclusions within one population or geographical area. For ex- 
ample, the number of telephone complaints received by a local au- 
thority may not be a reliable indicator in some underdeveloped or 
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impoverished areas because it presumes that citizens who wish to 
make complaints have access to a telephone. 


External reliability 

External reliability demands that the same conclusions be drawn from 
indicators, irrespective of the population’s size or location — that is, gen- 
eralizations can be made with these indicators. For example, using the 
number of students per 100 inhabitants as an indicator of the level of 
training of the inhabitants in a particular city could be highly mislead- 
ing, either because of the existence (or lack of) a university in the city or 
because of the degree of prestige and popularity of the universities in 
the city. So, a city of the size of Oxford (home of a university) but 
situated 50 km away will have many fewer students per 100 inhabitants 
than does Oxford, although the overall level of training of the inhabit- 
ants may not be very different. Another indicator would probably be 
required to measure the average training level of the population. 


Specificity 

An indicator is specific if its value remains stable when other data with 
a similar context change — that is, if it reflects only the parameter in 
question. The indicator must change when the parameter changes and 
must remain constant when the parameter remains constant. Being as- 
sured of specificity is difficult, mainly because so many factors (such 
as socioeconomic status and unemployment rate) interact with one an- 
other and are difficult to separate. For instance, life expectancy is not a 
specific indicator of good health or of long life. Life can be long, but 
very unhealthy. A more specific indicator would be the average life 
expectancy at the age of 40 and might include a list of disabilities. 


Sensitivity 

An indicator is sensitive if small fluctuations in the parameter are re- 
flected by small fluctuations in the indicator itself. Again, life expect- 
ancy is not a particularly sensitive indicator of health status because it is 
unlikely that small gains or fluctuations in overall health status are re- 
flected accurately. 


Relevance 

Indicators are said to be relevant if they relate to the appropriate 
data or phenomenon studied. Relevance is not as much a property 
of an indicator as a property of a group of indicators within a given 
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framework. Some indicators are thought to indicate ideas, which they 
in fact may not. For instance, the percentage of solid waste recycled 
in a city is certainly not an indicator of sustainability, although it 
might be argued that more recycling implies more sustainability; this 
is not the case because recycling by itself is not a particularly rel- 
evant indicator of sustainability. Nevertheless, combining recycling 
with the level of waste produced can provide a good measure of 
sustainability (less waste and more recycling is more sustainable). 


Easily obtainable 

Minimizing the burden on those providing data is desirable. Meas- 
urements are limited by such practicalities as cost and ease of col- 
lection. When the choice is available, it is better to collect data that 
are more easily obtainable than not. Sometimes “easily obtainable” 
is translated to “already collected,” but such an approach is too sim- 
ple. In order to follow the new development model and promote 
overall sustainability, simply relying on existing data by default is 
problematic. As mentioned earlier, sulfur dioxide is often used to 
indicate air quality because most localities, if they measure any air 
pollutants, measure sulfur dioxide. But just because data for sulfur 
dioxide are very common does not necessarily mean that they are 
the best indicator of air quality. Evaluators must make trade-offs 
among indicators that are readily available and those that are most 
desirable. The available indicators may not be exactly suitable for 
what is desired, but collecting additional information is not always 
feasible, owing to expense, political will and other factors. 


CoMPLEXITY OF INDICATORS 


Another important aspect to consider is the complexity of an indica- 
tor. Different categories of indicator with increasing complexity have 
been identified: simple numerical, complex numerical and qualita- 
tive indicators. Several examples of these indicators are provided in 
the sections that follow. 


Simple numerical indicators 

The simplest type of indicator is raw data expressed either in basic 
numbers (such as counts) or in rates of occurrence. Examples of 
basic numbers are the number of food inspections by a service, the 
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number of telephone complaints and the number of staff trained. 
Such raw data, however, are not as informative as rates of occur- 
rence, which combine different data sets. Examples of these rates 
include the number of service inspections per month, the number of 
fines issued per inspection, the percentage of families living below 
the poverty line and the amount of money spent per telephone com- 
plaint handled. 


Complex numerical indicators 

Lists or indices of indicators are appropriate for measuring more 
complex issues. For example, many indices have been developed to 
gain insight into how people value quality of life, measure physical 
impairment and assess psychological wellbeing. These are gener- 
ally derived from surveys and questionnaires administered in a for- 
mal way. Answers can occasionally be combined into a single meas- 
ure, such as the overall value of wellbeing. Sometimes the questions 
in such questionnaires and surveys are weighted differently, accord- 
ing to different criteria. An example of a complex indicator is pro- 
moted by the WHO Healthy Cities project (see also Table 6, page 75). 
A set of 53 indicators covering health, health services, environment, 
and sociodemographic characteristics is regularly reported by nearly 
50 European cities (3). 


Qualitative indicators 

Qualitative indicators are narrative descriptions collected from dis- 
cussions with small groups of people (focus groups), from key in- 
formants (politicians, service staff) and from professional observa- 
tions. Examples of qualitative indicators include not only percep- 
tions by health professionals of their job satisfaction (or of their use- 
fulness) in completing given tasks, but also general perceptions by 
people of how well a service meets their needs and demands. Quali- 
tative evaluation tools are discussed further in Chapter 4. 


WHO USES INDICATORS? 


People from different sectors, administrative levels and countries 
may approach indicators very differently. Experience has shown that 
an indicator can assume a different meaning, according to who re- 
quests or develops it. Technicians, researchers and other specialists 
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may request accurate, detailed and focused indicators related to their 
own field of interest. Each group interprets the indicator in terms of 
its own speciality or background. If, for instance, we take the number 
of kilometres of bicycle paths in a city, the department that pro- 
motes cycling (as opposed to car transport) is likely to be interested 
in the absolute figure, because the more paths there are, the more 
likely they are to be used. The department in charge of traffic safety, 
however, may be interested in the percentage of bicycle paths that 
are separated from other traffic. Furthermore, the department in charge 
of promoting tourism in the city may be interested in the number of 
kilometres of bicycle paths that go through natural areas and in the 
number of monuments of historical interest that can be reached by 
bicycle. Each of the following groups may need different indicators 
or may implement the same indicator in a different way. 


Local authorities 

Local authorities use indicators to demonstrate progress and pro- 
vide accountability to the local community, and to report to regional 
and national authorities. Indicators are often unique to a particular 
locality, though efforts to standardize indicators at national and re- 
gional levels are being advocated across the European Region by 
national governments and international bodies (such as the Euro- 
pean Environment Agency (EEA)). An example of how local au- 
thorities are using indicators is provided in the final case study in 
Chapter 6: Performance indicators in the United Kingdom (page 158). 


National authorities and international organizations 
National authorities and international organizations often view indi- 
cators in a similar way, and many of these organizations provide 
funds to local authorities through various mechanisms. Indicators 
are used to compare similar aspects of different cities or areas and 
help these organizations prioritize them for assistance. National au- 
thorities and international organizations also use indicators as a tool 
for orienting policies and advice, and as a warning system that sig- 
nals when a situation is evolving in a specific direction. 


Networks and groups 

Groups of people who communicate with one another for a com- 
mon purpose are often referred to as a network. City networks, inter- 
national health networks and networks of NGOs also use indicators 
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for at least two reasons: to measure the success of their advice and to 
provide members with tools to measure the efficiency of their rec- 
ommendations. Indicators from networks are commonly sectoral, 
although some networks have a very broad scope. 


The community 

The community itself is also interested in indicators. Very simple 
figures, such as the inflation rate and the number of people unem- 
ployed, can have a large impact on the community. Some newspa- 
pers have tried to classify cities in the world (or in different coun- 
tries) by using a mixture of life expectancy, surface area of green 
space available, gross national product (GNP) and level of pollution 
of the rivers as an indicator. This type of mixture may have little 
scientific meaning, but it appeals to communities — who prefer to 
live in a city at the top of the list than in one at the bottom. Although 
it is not their primary mission, international organizations could re- 
duce the use of scientifically less valid indicators by providing the 
press with some good indicators that speak well to the layman. 


CATEGORIES OF INDICATOR 


Indicators can be categorized in many ways. Jones (4), for example, 
recommends categorizing indicators on the basis of how closely they 
reflect the values that drive environmental policy (see Box 6). The 
categories used in this volume are based on substantive areas of 
focus. Of the many broad categories of indicator now in use, five 
will be discussed here: process indicators, environmental indicators, 
health indicators, socioeconomic indicators and urban indicators. 
Some practical examples of indicators now in use will be given. It 
must be noted that this list is far from being complete, but it does give 
the reader an understanding of the breadth of indicators available. 


Process indicators 

An evaluation may focus on the way a service works — the process 
of transforming inputs into outputs (see Fig. 5). Process indicators 
have been developed for several purposes, varying from recurrent 
indicators used on a regular basis, such as those used for manage- 
ment information systems, to incidental ratios developed within a 
specific evaluation framework. 
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Box 6. Primary, secondary and tertiary indicators 





Primary indicators report directly the changing condition of environmental 
quality in terms of something that we value. A primary indicator immediately 
communicates the status of an environmental attribute without extensive tech- 
nical interpretation. 


Secondary indicators provide information about a stressor that affects envi- 
ronmental conditions. A secondary indicator may not communicate the condi- 
tion of the environment unless it is accompanied by information explaining the 
link between the stressor and the magnitude of its impact. 


Tertiary indicators measure management activities that address an environ- 
mental problem. By themselves, tertiary indicators do not provide any informa- 
tion about the environmental conditions, but they may address particular con- 
cerns about the response of a government to an environmental problem or they 
may provide information about the costs of different management strategies. 





Source: Jones (4). 


In general, every type of indicator (single numerical, complex 
numerical and qualitative) can be used as a process indicator. Com- 
mon process indicators are those that relate to process inputs, out- 
puts and features. 


Input indicators focus on what goes “into” a service. Information 
on input, such as funds or personnel (number of staff), is usually 
easily obtainable. Where inputs cannot be expressed in monetary units, 
qualitative measures for developing indicators may be appropriate. 


Usually outputs are described in terms of certain activities and 
tasks, such as activities performed, direct tangible results, or prod- 
ucts of a service. Outputs should not be confused with more indirect 
results, such as the impact of a service. Again, both quantitative and 
qualitative approaches are useful here. For example, the output of a 
service could be a certain level of satisfaction of people using the 
service; such outputs are especially suitable for qualitative analysis. 


Process feature indicators provide information on how a service 
actually works. In some cases indicators are developed by combin- 
ing input and output information (such as costs and benefits, and 
sometimes they focus on specific procedures within a service or on 
accounts of how service staff work together. Again, both quantita- 
tive and qualitative approaches are useful. 
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Process indicators are useful tools for managers, because they 
provide valuable information about the way services work. This is 
especially true when an evaluation compares the actual process with 
the planned process or when process features are compared to spe- 
cific standards. Since services have a role to play in communicating 
with the public, the way a service operates in relation to the commu- 
nity is of increasing relevance in evaluations. Process indicators are 
highly useful for answering such questions as: 


e How does the community or the individual feel about the ser- 
vices provided? 

e Does the service meet the demands of the community? 

e Are community and individual concerns handled properly? 


Several examples of process indicators can be identified. For ex- 
ample, in the United Kingdom the Audit Commission acts as a watch- 
dog for fiscal accountability at the local level. Performance indica- 
tors for local services have been developed, and many of these could 
also be considered process indicators. The fifth case study in Chap- 
ter 6 of this volume discusses the United Kingdom Audit Commis- 
sion performance indicators in some detail. Several examples of pro- 
cess indicators appear there in Box 27 (page 162). 


Process indicators vary widely with the type of service performed. 
To develop project-specific process indicators, evaluators may rely 
on conceptual models of the service implementation. A simple model 
for a waste collection service is provided in Fig. 11. Based on this 
model, many different process indicators could be developed, in- 
cluding (note this list is far from exhaustive): 


e volume of wastes (by type) collected; 

e average time spent by crews at collection points; 

e number of days refuse not collected on schedule; 

e cost per citizen; 

e person-hours required for different aspects of the service; and 
e frequency of collection points per neighbourhood. 


Environmental indicators 
Hundreds of environmental parameters are routinely collected across 
the European Region. Monitoring environmental conditions is 
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Fig. 11. Simplified waste collection process model 
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Source: WHO Regional Office for Europe (5). 


particularly useful for identifying areas of concern, measuring change 
and setting priorities, but is also useful for evaluating progress, effi- 
ciency and effectiveness. 


Many groups conduct environmental monitoring. It is conducted 
at national, district and local levels by environment agencies, citi- 
zens’ groups, non-profit organizations and private companies. In 
addition, many international organizations monitor the environment 
at the international level, including WHO, EEA, the World Bank, the 
Organisation for Economic Co-operation and Development (OECD), 
the United Nations Development Programme (UNDP) and the United 
Nations Environment Programme (UNEP). Many of these inter- 
national groups are developing indicators to translate environ- 
mental data into useful information and to better understand the 
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interactions among environmental, institutional, health, social and 
economic variables. 


Environmental indicators are often measured in more than one 
way at different scales. OECD, for example, recommends that data 
be tracked from both a regional and a city perspective for certain air 
pollutants (Table 1). 


Environmental indicators are commonly divided into three types: 
those that characterize a stressor or pressure upon the environment, 
those that measure the current state or magnitude of the problem, 
and those that measure human responses or management activities 
undertaken to address an environmental problem. These categories 
correspond loosely to Jones’s (4) classification of primary, second- 
ary and tertiary indicators (Box 6). 


Some organizations focus on only one of these types, while oth- 
ers focus on several types. The TEPI (Towards Environmental Pres- 
sure Indicators for the European Union (EU)) project is an example 
of the former. TEPI is a long-term monitoring project coordinated 
by the Statistical Office of the European Communities (EUROSTAT). 
This project has identified six major pressure indicators for each of 


Table 1. OECD air quality indicators 


Pollutant Unit Regional status and trends City peak statistics 


Carbon mg/m? Annual max. 8-hour running average Annual max. 8-hour running average 
monoxide 


Lead mg/m* Annual average Annual average 

Nitrogen mg/m? Annual max. 24-hour average Annual max. 1-hour average 
dioxide Annual average Annual max. 24-hour average 
Ozone mg/m® Annual max. 1-hour average Annual max. 1-hour average 


Annual max. 8-hour running average Annual max. 8-hour running average 


Particulate mg/m? Annual max. 24-hour average Annual max. 24-hour average 
matter Annual average 

Sulfur mg/m? Annual max. 24-hour average Annual max. 1-hour average 
dioxide Annual average Annual max. 24-hour average 





Source: OECD (6). 
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the EU’s ten core environmental policy fields. A sample of these is 
shown in Table 2. 


Other groups use more complicated models that look at many 
aspects of the environment. For example, several United Nations 
groups and EEA follow the “DPSIR model” because it distinguishes 
among driving forces, pressures, states, impacts and responses 
(Fig. 12). According to EEA (8): 


The DPSIR Framework provides an overall framework for analysing envi- 
ronmental problems. It shows how Driving forces, such as industry and 
transport, produce Pressures on the environment, such as polluting emis- 
sions, which then degrade the State of the environment, which then Impacts 
on human health and eco-systems, causing society to Respond with vari- 
ous policy measures, such as regulations, information and taxes, which can 
be directed at any other part of the system. 


Table 2. TEPI project indicators focusing on 
environmental pressures (selected samples) 





Policy field? Name 





Climate change Emissions of carbon dioxide 
Emissions of methane 
Emissions of nitrous oxide 
Emissions of hydrofluorocarbons 
Emissions of perfluorocarbons 
Emissions of sulfur hexafluoride 


Ozone layer depletion Emissions of bromofluorocarbons (halons) 
Emissions of chlorofluorocarbons 
Emissions of hydrochlorofluorocarbons 
Emissions of nitrogen oxides by aircraft 
Emissions of chlorinated carbons 
Emissions of methyl bromide 


Dispersion of toxic substances Consumption of pesticides by agriculture 
Emissions of persistent organic pollutants 
Consumption of toxic chemicals 
Index of heavy metal emissions to water 
Index of heavy metal emissions to air 
Emissions of radioactive material 


a Indicators for the following environmental policy areas are not shown: air pollution, loss of 
biodiversity, resource depletion, urban environment problems, marine environment and coastal zones, 
water pollution, and waste. 


Source: EUROSTAT (7). 
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Fig. 12. The DPSIR model 
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Source: EEA (8). 


The health and environmental analysis for decision-making 
(HEADLAMP) project (Box 7) is an example of a joint international 
programme that has adapted the DPSIR model to explore possible 
links between environment and health. The project combines envi- 
ronmental indicators with socioeconomic and demographic data and 
dimensions of health impacts (Fig. 13). 


Box 7. The HEADLAMP project 


HEADLAMP is a joint project of the United States Environmental Protection Agency, 
UNEP and WHO. The project’s main objective is to provide decision-makers, 
environmental health professionals and the community with valid and useful infor- 
mation on the local and national health impacts of environmental hazards. 


Local HEADLAMP applications are envisaged for monitoring progress towards 
sustainable development. One of the project’s most important activities is the 
development of environmental health indicators to be used to quantify and 
monitor local situations. 


Within the HEADLAMP framework, environmental health indicators are defined 
as synthesized information about known environment-related diseases or con- 
taminants with known adverse health effects. In order to identify indicators 
appropriate to a specific setting, environmental and health data already avail- 
able are used, and are linked together. In addition, the project developed the 
driving forces, pressure, state, exposure, effects, actions (DPSEEA) model, 
based on the pressure—state—response model developed by OECD. Adjusted 
for environmental health, the DPSEEA model categorizes the different possible 
indicators in five stages (see Fig. 13). 








Sources: Corvalán & Kjellström (9); Pastides (10); Kjellström & Corvalán (11); Schwartz & 
Corvalán (12); World Health Organization (13). 
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Fig. 13. The conceptual framework for the development and 
identification of the HEADLAMP environmental health indicators 
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Source: Kjellström & Corvalán (11). 





















Each of the boxes in Fig. 13 can be paired with an environmental 
problem to develop a matrix of indicators. Two simple examples are 
provided in Table 3. 


Table 3. Examples of the HEADLAMP indicators 





Indicator Definition Examples 

type 

Driving forces The basic socio- Use of lead as an Production of sewage 
economic activities additive in petrol 
relevant to a specific used in cars 
area within 


environmental health 


Pressure Consequences of the Concentration of lead Quantity of sewage 
driving forces in terms found in petrol 
of environmental 


change 

State The environmental Concentration of lead Presence of faecal 
quality resulting from found in ambient air coliforms in surface 
the pressures exerted water 

Exposure The nature and level Number of houses Presence of faecal 
of interaction between estimated to contain coliforms in drinking- 
humans (or other lead paint water 


receptors) and the 
environmental state 


Effects The results (usually in Raised levels of lead Gastrointestinal disorders 
human terms) of in children 
changes to the Neurological impairment 


environmental state, 
based on the exposure 


Actions The wide range of Number of cars using Expenditure on waste 
possible responses leaded petrol management activities 
to prevent any of the Percentage of area Percentage of population 
above banning leaded with access to treated 


petrol drinking-water 
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Lead as a fuel additive is a useful example, because it has been 
well studied and because sufficient evidence for causal links is avail- 
able. In addition, Needleman and Kim et al. (14,15) have linked 
neurological impairment with certain levels of lead in human blood. 
Unfortunately, most environmental health issues are less well under- 
stood — causal links are less direct, exposures are based on models 
and default assumptions, or effective biomarkers (such as the level 
of lead in blood) are unavailable — making it difficult to develop the 
full matrix of indicators. 


Access to appropriate facilities and the ability to collect data over 
time are prerequisites for using environmental indicators in evalua- 
tions. Monitoring should be carried out over a sufficient period of 
time to identify and follow changes in environmental conditions. 
This requires a stable resource base and access to facilities and tech- 
niques. Facilities for collecting and analysing data are often located 
away from environmental health services themselves, in private labo- 
ratories or central environmental agencies. Such institutional arrange- 
ments complicate data collection and analysis efforts and lead to 
inconsistencies in data quality. Consistency and quality of data are 
influenced by financial resources, equipment and staff expertise, 
which vary widely across the Region. 


The use of environmental indicators is occasionally unnecessary 
for environmental health service evaluation because environmental 
and health impacts are only some of the possible evaluation sub- 
jects. When evaluating efficiency or relevance, for instance, pro- 
cess indicators may be more favourable than environmental moni- 
toring indicators. Also, environmental indicators are of little value in 
evaluating services addressing environmental health promotion. 


Health indicators 

There are also a multitude of health indicators available. Because 
health is multidimensional, many indicators have been developed 
that reflect various facets of health. Morbidity, mortality and life 
expectancy are among the most well known health indicators. These 
three indicators, however, may prove to be of little use in making 
decisions about environmental health services, because conclusions 
about the influence of environmental factors on morbidity, mor- 
tality and life expectancy can be drawn only in the rare cases when 
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sufficient data are available and when causal relationships are straight- 
forward. 


Other health indicators are becoming more useful. For example, 
health science has been focusing more and more on such measures 
as functional disability, which deals with the consequences of dis- 
ease on the individual and on a social level. Psychologists are de- 
veloping measures that reflect the personal perception of health as a 
multifaceted state, involving much more than disease and death 
alone. The social impacts of bad health can also be measured — for 
example, as the extent to which medical services are used or the 
average annual sick leave taken in a company. Predictors of health 
are used in preventive strategies — for instance, by identifying such 
risk groups as pregnant women, infants and the elderly. Though it is 
hard to avoid classic mortality figures, doing so is essential if all 
facets of health are to be covered when evaluating environmental 
health services. 


Among health indicators, we find both the traditional measures 
of ill health and the more recently introduced measures of health 
status. Donaldson & Donaldson (/6) summarized six types of health 
indicator as critical elements in a comprehensive framework for health 
needs assessment. The six categories cover health status, people, 
health services, population, disease and mortality (Table 4). 


Health indicators are collected by international, national and lo- 
cal organizations. On an international scale, the WHO Regional Of- 
fice for Europe collects health statistics from around the Region. 
National ministries of health also collect and maintain disease regis- 
ters and relevant statistics. Local health offices, hospitals, health man- 
agement organizations, occupational health clinics and private health 
care facilities are additional sources of health data. When health data 
are obtained from local sources, confidentiality is an important con- 
sideration. A patient’s privacy should be protected by structuring 
health information so as to prevent the identification of a specific 
individual. 


Analysing behavioural patterns (such as the number of people 
who take public transport or the solid waste produced per house- 
hold) is an important way to find relationships between environment 
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Table 4. Health indicator categories 


Description 


Examples 





Health status 
(see also Box 14, 
page 99) 


People 


Health services 


Population 


Disease 


Mortality 


Indicators that relate to health 
status do not focus on such 
things as disease, but do 
focus more on aspects of 
wellbeing, quality of life and 
the individual’s perception of 
his or her own health. 


By gathering information on 
lifestyles and other individual 
features, one can acquire a 
better understanding of the 
factors that determine health 
and wellbeing. 


Information on (environmental) 
health service activity is also 
relevant to acquiring a better 
insight into the health of 
people. Health problems, 


Pain/wellbeing 

Social functioning 

QALYs (quality-adjusted 
life-years) 


Lifestyles 

Risk factors 
Views/experiences 
Drug consumption 


Treatment thresholds 
Uptake 
Access 


however, do not always extend 


to the level where care or 
treatment is needed. 


As a baseline measurement 
of the health of a specific 
population, it is essential to 
have information on the 
specific population structure. 


The traditional measures of ill 
health and disease are still 
valid, though it is commonly 
agreed that they only present 
one dimension of the nature 
of health and wellbeing. 


As with disease indicators, 
mortality only presents a 
limited picture of actual ill 
health status. More elaborate 
indicators of disease and 
death can be useful in some 
instances. 


Size 
Composition 
Changes 


Incidence/prevalence 
Incapacity 
Survival 


Mortality rates by cause 
Avoidable deaths 





Source: Donaldson & Donaldson (16). 


and health. These examples of behavioural patterns may not measure 
direct impacts on health, but they can indicate positive (or negative) 
momentum. A mix of quantitative and qualitative approaches may be 
useful for reaching a better understanding of the links between environ- 
ment and health and the influence of an environmental health service. 
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Socioeconomic indicators 

Organizations are increasingly using socioeconomic indicators be- 
cause environmental health also consists of the social and economic 
circumstances that affect health. Relying on data from WHO, the 
World Bank, UNDP and the United Nations Economic Commission 
for Europe (ECE), WHO used the following socioeconomic indica- 
tors in its report Health in Europe 1997 (17): 


e midyear estimated population (millions) 

e GNP per person 

e gross domestic product (GDP) per person 

e total health expenditure as a percentage of GDP 

e mortality per 1000 live births 

e life expectancy at birth (in years) 

e human development index? 

e immunization coverage‘ 

* unemployment as a percentage of the labour force. 


Indicators that relate social or economic developments directly 
to health status (for example, Box 8) are especially interesting, al- 
though much work in this area remains to be completed. 


The United Nations has been developing indicators that relate 
socioeconomic indicators to health and environmental concerns. The 
United Nations has used a simplified version of the DPSIR model 
(see Fig. 12) to develop a matrix of indicators for sustainable devel- 
opment (19). For each major category, three types of indicator are 
presented (whenever possible), including: 


e driving force indicators — to capture information about hu- 
man activities, processes and patterns; 

e state indicators — to capture information about the state of hu- 
man development; and 


3? The human development index measures the average achievements of a country in 
three basic areas of human development: longevity, knowledge and standard of living. It 
is a composite index of three variables: life expectancy, educational attainment (adult 
literacy and combined primary, secondary and tertiary enrolment) and real GDP per 
person. 


* Percentage of children at 12 months of age who received their third dose of oral 
poliomyelitis vaccine. 
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Box 8. Social status and asthma 





In 1995, a group of Canadian researchers set out to study the relationship 
between socioeconomic status and indicators of asthma in children. The socio- 
economic status was assessed by studying the occupation of the parents. 
Asthma was analysed by using exercise-induced bronchospasm (EIB). Data 
on this condition were available for 989 children. In addition, information on 
home exposure to tobacco smoke, demographics and respiratory symptoms 
was gathered through a questionnaire. 


The results were presented using odds ratios, complex indicators that estimate 
the relative risk that results from various factors. An odds ratio (OR) of < 1 
implies that, under the conditions of the analysis, the researchers were unable 
to observe a difference in the two groups. Compared to children from the most 
advantaged homes, children from the least advantaged homes were more 
likely to have EIB and to report night cough and cough with mucus (see odds 
ratios below). 


Odds ratios for the children from the least advantaged homes compared 
to those from the most advantaged homes 


EIB OR = 2.26 
Reporting night cough OR = 2.30 


An OR of > 2 means that children from poorer homes were more than twice as 
likely to have EIB and report night coughs, under the conditions of analysis. 


Odds ratios for EIB in relation to other exposure factors 


Excess EIB related to the presence of a cat at home OR = 1.63 
Excess EIB related to respiratory infection during infancy OR = 1.71 


The results did not show significant reporting of wheeze or diagnosed asthma 
among disadvantaged children but the researchers demonstrated that, through 
the use of social and health indicators, a clearer picture of the social dimension 
of health can be produced. Based on the results, the researchers suggested 
that unidentified environmental factors contribute to the excess asthma mor- 
bidity in poor children. 








Source: Ernst et al. (78). 


e response indicators — to capture information about policy op- 
tions and other human responses. 


A selected set of the United Nations’ working list of indicators 
for sustainable development is presented in Table 5. 


Urban indicators 

Since the majority of the world’s population now resides in urban 
areas, our final area of focus is urban indicators. Both national and 
international organizations (such as WHO and OECD) are increasingly 


DATA AND INDICATORS 


73 


Table 5. Working list of United Nations sustainable development 
indicators (selected examples) 





Chapters of 
Agenda 21 


Demographic 
dynamics and 
sustainability 
(Chapter 5) 


Promoting 
education, 
public 
awareness 
and training 
(Chapter 36) 


Protecting and 
promoting 
human health 
(Chapter 6) 


Changing 
consumption 
patterns 
(Chapter 4) 


Financial 
resources and 
mechanisms 
(Chapter 33) 


Transfer of 
environmen- 
tally sound 
technology, 
cooperation 
and capacity- 
building 
(Chapter 34) 


Driving force 
indicators 


Population growth rate 
Net migration rate 
Total fertility rate 


Rate of change of 
school-age population 
Primary and secondary 
school enrolment 
ratio (gross and net) 
Adult literacy rate 


Annual energy 
consumption 

Share of natural- 
resource-intensive 
industries in value- 
added manufacturing 


Net resource transfer/ 
GNP 
Total overseas 
development 
assistance given 

or received as a 
percentage of GNP 


Capital goods imports 
Direct foreign invest- 
ments 


Source: United Nations (19). 


State 
indicators 


Response 
indicators 


Population density 


Children reaching 
grade 5 of primary 
education 

School life expectancy 

Difference between 
male and female 
school enrolment 
ratios 

Women per hundred 
men in the labour 
force 


GDP spent on education 


Basic sanitation Immunization against 
Percent of population infectious childhood 
with adequate excreta diseases 

disposal facilities Contraceptive prevalence 
Access to safe Proportion of potentially 
drinking-water hazardous chemicals 
Life expectancy at birth monitored in food 


Lifetime of proven 
energy reserves 
Intensity of material 
use 

Share of manufacturing 
value-added in GDP 
Share of consumption 
of renewable energy 
resources 


Debt/GNP 
Debt service/export 


Environmental protection 
expenditures as a 
percentage of GDP 
Amount of new or 
additional funding for 
sustainable 
development 


Share of environmen- 
tally sound capital 
goods imports 


Technical cooperation 
grants 
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developing and using urban indicators to gain a better understand- 
ing of urbanization and its effects. Indicators that relate urban con- 
ditions to health status may prove to be especially useful: they in- 
clude indicators that account for urban issues, such as social condi- 
tions (employment, crime, violence), housing, mobility and 
sustainability at the city level. Urban indicators, however, are par- 
ticularly difficult to develop, as the complex system of a city is hard 
to translate into (a group of) numbers or indicators. In addition, in- 
dicators are sometimes vastly different from one city to another, which 
complicates efforts to compare them. There is a clear need for a com- 
mon denominator for indicators — a standard set used by a wide range 
of cities — to improve the comparability and possibilities for evaluation. 
An example of sustainability indicators at the city level is provided by 
the Sustainable Cities project of the European Commission (20). 


The WHO Healthy Cities project has developed a number of in- 
dicators to measure different aspects of urban health. A selection of 
these is presented in Table 6. 


The sample indicators provided in this chapter vary in their for- 
mat and presentation. Some are rather vague (such as those found in 
Table 4) and others are more specific (such as those found in Ta- 
ble 5). Others, such as those in Table 6, are quite explicit and in- 
clude not only the name of the indicator and a description, but also 
an exact formula that can be used to perform any calculations re- 
quired and the units to be used for its expression. Providing this 
level of detail enhances the internal reliability of a data set; having 
clear data definitions helps to ensure that the data are consistent 
across respondents. Such standardization is important if indicators 
are to be meaningful. 


DATA AVAILABILITY 


The absence of data can present a formidable challenge to evalua- 
tors — without data, change cannot be measured and comparisons 
cannot be made. Appropriate data may be unavailable for a variety 
of reasons; for example, comprehensive monitoring methods may 
not be feasible in some locations due to insufficient resources, and 
questionnaires may generate only limited responses. Also, data may 


Table 6. WHO Healthy Cities project urban indicators (selected sample) 





Indicator Description Calculation Units 

Public access to green spaces This indicator shows the surface area of Total number of m? of green spaces with public access m°? per inhabitant 
green spaces per inhabitant open to the Number of inhabitants 
public 

Derelict industrial sites This indicator shows the percentage of Surface area of derelict industrial sites x 100 Percentage 
derelict industrial sites compared to the Total surface area of the city 


total surface area of the city 





Pedestrian streets This indicator shows the importance Total length of pedestrian streets km/km? 
accorded to pedestrian streets Surface area of city 

Cycling in city This indicator shows the importance Total length of paths reserved for cyclists km/km? 
accorded to bicycle paths Surface area of city 

Public transport This indicator shows the number of seats Average daily number of seats x 1000 Seats per 
on public transport per 1000 inhabitants. Total population 1000 people 

Public transport network cover This indicator shows the number of Total number of kilometres served by public transport x 100 Percentage 
kilometres served by public transport Total number of kilometres of streets 


compared to the total number of 
kilometres of streets in the city 


Living space This indicator shows the average number Total number of rooms Number of 
of rooms per inhabitant Number of inhabitants rooms per 
The rooms are counted if they have a inhabitant 


distinct purpose or if they are > 4 m°, 
e.g. kitchen, dining room and bedrooms. 
Bathrooms, laundry rooms, hallways, etc. 
are not counted as rooms 





Source: Webster & Price (3). 
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be inadequate when scientific theories evolve; for example, it is widely 
accepted that a new development model is required to integrate en- 
vironmental, health and economic sectors, and to incorporate new 
sets of values, perceptions and indicators. The data and correspond- 
ing organizational structure available today may not reflect these new 
ideas and values. Existing data, which reflect the ideas of the past 
10-20 years, should be examined carefully to ensure that new data 
and structures are developed (when necessary) to keep pace with 
changing values and ideas. 


Ideally, evaluators tailor data collection and analysis specifically 
to each evaluation. Data collection, however, can be very expensive, 
and environmental health service decision-makers commonly rely 
on data sets that have been collected for other purposes. For exam- 
ple, managers in a local district may use data sets that have been 
collected separately by individual towns or cities in the district. This 
is a strategy that should be used with caution because data may not be 
comparable from one place to another. Efforts to fully define the prob- 
lems and uncertainties inherent in the data are particularly important 
when they are used for purposes other than those originally intended. 


Another way to handle a lack of data is to generate information 
with models. Extrapolation models can be developed to extend em- 
pirical data, and to make predictions when no data are available. 
Using models adds uncertainty, but they are important tools for fill- 
ing gaps in the available data. 


When data are particularly difficult or expensive to collect, proxies 
or surrogates can be used. Proxies are essentially indicators of other 
indicators (2/). For example, traffic density or city petrol consump- 
tion could be used as proxies for air quality. Although characteristics 
of the air itself are not measured directly, a major cause of poor air 
quality (driving cars) is measured. Another example of a proxy indi- 
cator is monitoring cigarette sales to approximate tobacco smoke 
exposure. 


Even when a large body of knowledge exists, data might not fa- 
cilitate decision-making. Data may be overwhelming, too technical, 
contradictory, or presented in a format that does not allow the 
straightforward conclusions that decision-makers prefer. Collecting 
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additional data is sometimes the best course of action. Chapter 4 
deals with the tools used to gather data and measure change. 


RECOMMENDATIONS AND PRECAUTIONS 


Indicators are developed to indicate; they are not exact measures of 
change. Evaluators must recognize that indicators are at least partly 
inaccurate. It is almost impossible to develop an indicator that has all 
of the desirable objectives mentioned in this chapter. Those who use 
and develop indicators should clearly identify the aspects of an indi- 
cator that are less accurate — that is, define uncertainty as clearly as 
possible. This will help prevent organizations or individuals from 
erroneously using indicators as exact measures. 


Avoid heavy reliance on single figures to summarize complex 
situations. In the past, there has been a tendency to present the health 
status of a population by using mortality or life expectancy figures 
alone. Such rough indicators can mask important details. (For exam- 
ple, increasing the quality of life for elderly people in the years just 
before death is not reflected in life expectancy statistics.) Thus, it is 
recommended that a series of indicators be used to characterize the 
rich and complex concepts of sustainability and quality of life. 


Similarly, avoid simple designations for programme success or 
programme failure. Such conclusions are unsatisfactory, because not 
enough detail is available to provide the feedback needed to make a 
decision or to implement and improve future services. More detail 
can be presented by developing accurate groups of indicators. 


A team of involved parties should decide which indicators are to 
be used in a particular situation. Although involving interested and 
affected parties (stakeholders) in the selection of indicators may ini- 
tially require more effort, the process is likely to be fairer because 
more points of view are considered. For example, a management 
team assisted by technical experts and citizens’ representatives might 
agree on a core set of indicators for tracking air pollution in a local 
district. The process of agreement might be difficult because of con- 
flicts of interest over many issues, such as the purpose of the indica- 
tors, the number of indicators to use, the best type to use, whether to 
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use existing information or to collect new data, and the cost of the 
indicators. Once a balance among the interests is sought and a gen- 
eral consensus about the indicators is reached, the management proc- 
ess should improve. 


A multidisciplinary approach to choosing appropriate indicators 
is often necessary because the systems and elements to be measured 
are often highly complex. Epidemiologists, economists, statisticians 
and social scientists can be vital to the process of developing and 
interpreting indicators. 


The primary purpose of developing and using indicators is to 
allow environmental health service managers to move towards over- 
all sustainability in a healthy way. Complete data are never available 
and policy-makers often have to make decisions based on insuffi- 
cient evidence. The absence of data, however, should never be used 
as an excuse for inaction. 


Developing indicators to track service performance can lead to 
unintended results. For example, once indicators have been estab- 
lished, service managers and staff may be tempted to direct their 
efforts towards improving the outcomes for that particular indicator, rather 
than towards the overall improvement of service delivery or outcome. 


The power of indicators is limited because they may be inter- 
preted differently and, so, do not oblige managers and decision- 
makers to act. 


Box 9 summarizes the key recommendations of this chapter. 


Box 9. Key recommendations from Chapter 3 
Assess the quality of data available; assess the need for more data for evalua- 
tion; and always state clearly the limitations of data used 


State (explicitly) how to deal with uncertainties — despite the usefulness and 
necessity of models, estimates and projections in supplementing data 


Consider using indicators in the evaluation process; be aware that their quality 
depends on a number of basic properties 


Consider using different indicators to describe different phenomena at the same 
time because indicators indicate — they are not exact measures 
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Instruments of Evaluation 


Previous chapters describe evaluation as a process for com- 
paring information about programmes, geographical areas, 
people and time, so that improvements in environmental health 
services can be implemented. Information is essential for meas- 
uring change, making comparisons and drawing conclusions, as 
part of evaluation activities. This chapter addresses several 
methods and tools for gathering data, translating it into useful 
information and drawing conclusions. Particular attention is paid 
to economic evaluation and qualitative evaluation tools. 


MAKING COMPARISONS 


The third major step in the evaluation plan (Fig. 4) is to conduct the 
evaluation. Three separate but interrelated activities are identified: 
choosing a methodology, assessing the situation and drawing con- 
clusions. Underpinning these activities is the need to make com- 
parisons. Commonly these comparisons are based on indicators (see 
Chapter 3), which provide standard measures upon which to make 
judgements, set goals and measure progress towards those goals. 
Three types of comparison have been introduced earlier (Fig. 2): 


e the present situation is compared with the past situation; 
e the present situation is compared with another similar situa- 


tion; and 
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e the present situation is compared with a predetermined 
standard. 


The first type of comparison, comparing the present situation 
with a situation(s) in the past, generally means that similar data points 
are collected and observed over a period of time. This strategy is 
essential for understanding changes over a period of time (trends). 
Trend analysis can involve both quantitative and qualitative data. 
Time-series analysis is used increasingly in such applications as meas- 
uring peaks and trends in pollutant levels, predicting health effects 
and exposures, and evaluating new monitoring equipment (/). Trend 
analysis, however, is limited because data need to be collected con- 
sistently over a period of time and because situations need to be 
fairly similar for comparisons to be valid. These requirements are 
difficult to ensure over a span of time. For instance, monitoring tech- 
niques and equipment evolve with time and calibration of these de- 
vices can be difficult (see also Box 10). 


The second type of comparison measures one situation and com- 
pares it with another situation. This practice is quite common: data from 
one area are compared with those from another area, or service agen- 
cies working towards the same goal are compared to each other (also 
known as benchmarking, see Box 11). An important obstacle to this 


Box 10. Comparisons over long periods of time 


Comparison is an essential component of any evaluation, but evaluators should 
exercise caution. In order to make meaningful judgements, it is important to look 
at the context of the situations being compared, to understand the degree of 
similarity or dissimilarity. In particular, changes with time may be difficult to 
compare because strategies for data collection evolve with time. When major 
evolutions occur, comparisons may be more difficult. 


Restructuring the International Classification of Diseases (ICD) is an example 
of a major change that could invalidate a comparison over a span of time. The 
ICD is used by medical professionals on death certificates to classify a pa- 
tient's primary cause of death. Periodically, the ICD is updated to reflect emerg- 
ing medical knowledge. New categories are added, others are renumbered and 
still others are removed altogether. 


When changes are made, previous records are not reviewed or recoded. The 
resulting incompatibilities prevent straightforward comparisons of death statis- 
tics over long periods of time. Although this example comes from the medical 
discipline, it serves to illustrate a problem that is also common in environmental 
health codes and statistics. 
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type of comparison is that specific circumstances can differ mark- 
edly from one geographical area to another (and over time), which 
potentially makes comparison less valid or fair. For example, it would 
probably be unwise to compare regional service activities directly 
with activities at the local level, because levels of financial resources, 
personnel and other factors would be too dissimilar. 


The third type of comparison uses the data and information gath- 
ered and compares them with predetermined standards. For exam- 
ple, data are often compared with specific quality standards, such as 
those put forth by ISO, by engineering institutions in codes of prac- 
tice and by legislation. Standards are often set as targets (or goals) 
for a policy when it is written. These might include, among others, 
political, social, economic and moral standards. When predetermined 
standards are used to evaluate policies, there is a risk of ignoring 
unexpected and unwanted effects of that policy. “Goal-free evalua- 
tion” (3,4) is proposed as an alternative strategy to circumvent such 
problems. The goal-free approach advocates comparing results with 
an expressed set of needs, as opposed to limiting the evaluation to 
examining the intended impacts. 


Developing conclusions 

The final activity in an evaluation is to interpret the results. Conclu- 
sions are often developed by answering the questions raised before 
and during an evaluation process. Evaluators should be particularly 


Box 11. Benchmarking 





Originally, benchmarking was an approach developed within large companies 
to improve business performance, but it can also be tailored to environmental 
health services. The approach is relatively simple. 


To begin, a “benchmark performance” — a standard for certain aspects of 
service delivery (such as the maximum number of complaints received from 
the public per month) — is defined, based on existing performance. This bench- 
mark can be derived from various sources, such as the local government, the 
community (the service clients) and a number of similar services. 


Once a benchmark is set, other services try to implement the practices needed 
to achieve the benchmark, or go beyond it. The goal behind benchmarking is to 
improve service delivery, by learning from other experiences, and to build on 
knowledge already available. 





Source: United Kingdom Department of Trade and Industry (2). 
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careful when extrapolating or generalizing, and when thinking about 
causality. Because evaluations are designed uniquely for the pur- 
pose at hand, no single strategy for developing conclusions can be 
provided. The techniques that will be discussed involve different 
strategies for developing valid conclusions, and each has its own 
strengths and weaknesses. 


EVALUATION METHODOLOGY 


Once the type of comparison has been identified, the evaluation 
methodology can be selected. Four broad categories of methods are 
particularly important for evaluating environmental health services: 
environmental monitoring, health monitoring, economic evaluation 
and qualitative evaluation. In the past, these have often been classi- 
fied either as quantitative or as qualitative (the former being associ- 
ated with numerical or statistical analyses and deductive reasoning, 
and the latter being associated with narrative description and induc- 
tive reasoning). Such distinctions are simplistic and, generally, should 
be avoided. The terms qualitative and quantitative should not be 
seen as opposites, but should instead be seen as complementary 
approaches that rely on different kinds of information and reason- 
ing techniques. 


In the sections that follow, environmental monitoring, health moni- 
toring, economic evaluation and qualitative evaluation are discussed. 
This volume places a major emphasis on economic and qualitative 
evaluation methods because the literature on the subject has not con- 
nected these topics to environmental health services evaluation. In 
contrast, much has been written on environmental monitoring (5) 
and health monitoring (6,7), so these issues are covered only briefly 
in the sections that follow. All the methods discussed in this chapter 
have unique strengths and weaknesses. Thus, no single method alone 
is appropriate for evaluation. A combination of the methods pre- 
sented here is recommended for robust evaluation of environmental 
health services. 


Environmental monitoring 
Environmental monitoring is the practice of measuring aspects of 
the physical environment on a regular basis (see also Chapter 3 for a 
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discussion of environmental indicators used in monitoring). Many 
environmental factors are not yet measurable in a reliable manner or 
at an affordable cost, so the number of environmental parameters, 
substances and compounds monitored (for example, sulfur dioxide 
in the air or nitrates in the water) is limited. Generally, monitoring is 
also limited to a specific area, for example, a community, a metro- 
politan area, a country or a region. Nevertheless, environmental 
monitoring can be used to measure several important aspects of the 
environment, such as those that follow. 


e Change: Are environmental parameters changing with time? 

e Progress: Is environmental quality improving? 

e Impacts: What are the changes in environmental quality that 
result from policies or service activities? 


For several reasons, environmental monitoring alone may be in- 
sufficient for evaluating complex environmental health services and 
actions. First, many evaluations are concerned with the process of 
service delivery, or the relevance of the service to the needs of the 
population — environmental monitoring is unlikely to contribute 
significantly to such analyses. Even when impact analysis is the 
subject of an evaluation, weak causal relationships between envi- 
ronmental parameters and health effects prevent the prediction 
of exact population outcomes based only on contaminant con- 
centrations. 


Second, consistency can be difficult to achieve because data from 
different organizations and countries (or even from different areas 
within the same country) can vary widely. A wide range of organiza- 
tions, including local authorities, national agencies and international 
networks, monitor the environment (see also Fig. 10). Each of these 
groups is a potential source of environmental data. Among other fac- 
tors, inconsistencies result from differences in the kind of data col- 
lected, the information systems used to store data, and the techniques 
used for data collection, measurement and analysis. 


Third, the reliability of environmental monitoring data is also dif- 
ficult to ensure. Data can be unreliable if standards for laboratory 
practices and data collection processes are not followed, and if equip- 
ment is old or poorly maintained. 
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These limitations can be overcome to a certain extent by: 


e developing a minimum set of commonly used indicators; 

e linking environmental and health monitoring data together (see, 
for example, Box 7, which discusses a model being used to 
link data about health and the environment); 

e setting and meeting international standards (such as ISO stand- 
ards); and 

e developing close international and interorganizational coop- 
eration and harmonization. 


It is also recommended that during evaluations other tools and data 
be used simultaneously with environmental monitoring techniques. 


Health monitoring 

Health monitoring is the process by which different aspects of the 
health status of individuals (or a population) are measured on a regu- 
lar basis. As the promotion of a healthy general population is the 
basic objective of virtually all environmental health services, in many 
cases it makes sense to use health monitoring as a tool for evaluat- 
ing these services. 


There are numerous ways to gain insight into health status (see 
also Chapter 3 for a discussion of health indicators used for moni- 
toring). Traditionally, the field of epidemiology has been the source 
of data collected for such indicators as morbidity and mortality. Epi- 
demiology has also been useful in identifying risk factors surround- 
ing health problems. Various dimensions of health status, however, 
may not be measured directly through such techniques, because there 
is much more to health monitoring than epidemiology. Many other 
data sources can be used to study (changes in) health status; for 
instance, registers that store information about individual cases of a 
disease are often kept by local hospitals and health services. A 
number of examples of data-collection tools that monitor the health 
of individuals, groups or populations (7) are: 


e registration records (of disease, death, hospitalization, birth) 
e event (disease outbreak, accidents, sick leave) records 

e medical and pharmaceutical consumption records 

e environmental health service records 
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e notification registers 
e questionnaires 
e sentinel health event registers (see Box 12). 


When health monitoring is of sufficient quality, it is particularly 
useful in assessing evaluation areas oriented towards service out- 
puts, such as impact, efficiency, effectiveness, adequacy and 
sustainability. Combining health data with other types of informa- 
tion (such as demographics, costs, geographical distribution and so- 
cial factors) can enhance the usefulness of health data in assessing 
more complicated subjects, such as equity and effectiveness. 


The use of health data for the evaluation of environmental health 
services is, however, limited. Health monitoring data are less useful, 
for example, when the subject of an evaluation is process, relevance 
or equity, because the population outcome is not the main concern. 
In addition, health monitoring data can sometimes be of poor qual- 
ity, just as environmental data can be of poor quality. Health data 
may be biased (for example, by social or economic differences), or 
they may simply be insufficient (for example, collected over peri- 
ods too short to observe change). 


Furthermore, health status may not always reflect specifically and 


reliably the performance of services. Many causes and effects pre- 
cede changes in health status, both for individuals and the general 


Box 12. Occupational sentinel health events 





An occupational sentinel health event can be defined as a disease, disability or 
untimely death that is occupationally related and whose occurrence may: 


provide a stimulus for epidemiological or industrial hygiene studies; and 
e serve as a warning signal that material substitution, engineering control or 
medical care may be required. 


When such events are registered in centralized locations, large amounts of 
information on possible occupation-related diseases and other health effects 
can be gathered and stored. The resulting lists of relevant occupational sentinel 
health events (registers) are extremely helpful in monitoring occupational health 
and identifying priorities or problem areas. 





Source: Mullan & Murthy (8). 
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population. Since the relationship between environment and health 
is complex, it is difficult to model and predict. Adding the actions of 
a service to the equation further complicates the relationships. For 
instance, many environmental health services try to prevent nega- 
tive health outcomes or, in other words, to produce a non-event (see 
also Chapter 2). An example of this is the prevention of food poisoning, 
where the absence of food poisoning cases is considered a success. 
How are evaluators to know whether a low number of food poisoning 
incidents results from protective environmental health services or from 
other conditions over which the service managers have no control (such 
as unseasonably low temperatures that prevent microbial growth)? 


Economic evaluation 

Economic evaluation can be defined as the comparative analysis of 
alternative courses of action in terms of both their costs and conse- 
quences. The aim of an economic evaluation is to provide 
stakeholders (such as policy-makers, service staff, affiliated organi- 
zations and the community as a whole) with information about the 
issues of efficiency and effectiveness of a service. Economic evalu- 
ations are typically categorized as either partial or full economic 
evaluations (see also Table 7). 


Partial evaluations look only at either inputs or outputs. 


¢ Inputs. Some analyses may simply include the inputs of a ser- 
vice, such as the funding. This is called cost analysis (when 
using quantitative data) or cost description (when using more 
qualitative data). Strictly speaking, this is not a type of eco- 
nomic evaluation, but is instead a financial analysis. Different 
input alternatives, however, are compared to each other, and 
an evaluation component is introduced. The aim is then to 
pick the alternative with the most appropriate cost. 


e Outputs. Benefit analysis (or description) is an account of the 
effects of a service, and is thus only a partial economic evalu- 
ation for comparing different outputs. 


Analysing only inputs or outputs does not tell whether services 
are efficient or effective, but partial evaluations are still often useful 
to decision-makers. 
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Table 7. Forms of economic evaluation 


Evaluation of: 


costs or benefits 





costs and benefits 





Costs Benefits 
One Partial evaluation Partial evaluation 
alternative 
Cost analysis or Benefit analysis or Cost—benefit description 
description description 
More Partial evaluation Full evaluation 
alternatives 
Cost evaluation Benefit evaluation Cost—effectiveness analysis 


Source: Drummond et al. (9). 


Cost-benefit analysis 
Cost—minimization analysis 
Cost-utility analysis 


Full economic analyses focus on inputs compared to outputs (or 
outcomes). They result in a conclusion about the efficiency of a ser- 
vice — that is, whether the outputs are reached at a reasonable ex- 
pense. Complete evaluations can be used to assess a single type of 
input and output, or they can be used to assess several alternative 


courses of action. 


Three key aspects are important in evaluating costs and ben- 


efits (9). 


1. Time. An economic evaluation can be performed before a 
project or plan is completed; this often assists decision-makers in 
picking the most efficient alternative course of action. An eco- 
nomic evaluation is also used after a project or activity so 
decision-makers can decide whether the activity is worth con- 
tinuing, or where adjustments should be made. Moreover, an 
economic evaluation is often conducted during an activity, 
when its goal may be to further improve the activity or reallo- 


cate resources. 


2. Information included in the analysis. As described earlier, dif- 
ferent information can be brought into the analysis to reach a 
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conclusion. More specifically, it is necessary to identify which 
inputs and outputs are taken into account. The sections that 
follow focus on the evaluation tools that simultaneously take 
inputs and outputs into account. 


3. Type of comparison. Economic evaluation can compare sev- 
eral alternative courses of action as well as compare a case in 
the present with one in the past. Sometimes a predetermined 
standard can be used to assess whether a service is working 
efficiently. 


General limitations to economic analysis 

Economic analyses are often used to evaluate services because a 
common, comparable and easily understood metric (money) is used 
as the basis for comparison. The use of economic evaluation tools, 
however, is limited by several factors. 


First, health as such has no monetary value (or price). Some as- 
pects of life can be given a monetary value — including a person’s 
projected earnings through labour, the cost of treatment, the cost of 
personal suffering, and the cost of suffering to family or friends — 
but even so, individual preferences for certain aspects of health may 
vary widely. Second, people may value money differently, depend- 
ing on their individual circumstances (for example, the value of a 
dollar saved may not be the same as that of a dollar spent). Third, 
natural resources (clean air, soil and water) are distinct from many 
other economic goods. Not only are they often limited and irreplace- 
able, but they are also without a market that treats them like a com- 
modity. Therefore, they are particularly hard to value in monetary 
terms. 


Furthermore, promoting environmental health implies that health 
promotion activities and environmental protection activities be put 
in place at the same time. These activities may not be compatible. 
For example, insecticides may be sprayed on marshlands to keep 
mosquitos from breeding and transmitting malaria. This practice is 
clearly advantageous from a public health perspective, but it may 
also produce a negative impact on the environment. Careful atten- 
tion is needed to trade-offs among environmental and health-related 
costs and benefits. 
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Steps in the economic evaluation process 
Five important steps in an economic evaluation can be distin- 
guished (9): 


. identification of alternatives 

. identification and measurement of inputs 

. identification and measurement of outputs 

. dealing with uncertainty and time preference 
. reaching a conclusion. 


aA BWN 


1. Identification of alternatives 

Ideally, an economic evaluation starts with identifying the possible 
alternatives, such as programmes, interventions or other activities. 
Developing a clear idea of which parts of a project are to be evalu- 
ated is critical. This step addresses the question: What should be 
evaluated? Problems may occur at this stage, as it is not always obvi- 
ous what alternatives are available or what (expected) costs and ben- 
efits are associated with each alternative. Sometimes one course of 
action is compared to taking no action at all, in order to show how 
an activity may impact the current situation and whether it is worth 
while starting or continuing that activity. 


2. Identification and measurement of inputs 
The next step in the process is to identify and measure inputs, very 
often costs. Costs involve a wide range of possibilities, including: 


e direct costs, such as costs for the service in question; 

e indirect costs, such as the costs to patients, their families, em- 
ployees, tax payers or the community as a whole; and 

* opportunity costs, resulting from not investing money in other 
activities. 


Costs should not simply be limited to direct costs in monetary 
terms. Some service inputs, such as the use of volunteer staff, may 
be hard to estimate but are highly valuable. 


3. Identification and measurement of outputs 

The next (and probably most complicated) step is identifying and 
measuring benefits or outputs. Fig. 14 gives some examples of the 
inputs and outputs relevant to environmental health services. 
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Fig. 14. Costs and benefits of an environmental health service 


Initial situation | - Input Activity Output Outcome | 

















Costs Benefits 

Organizing and operating costs (time, Changes in physical, social or emotional 
supplies, equipment, capital costs) functioning 

Costs borne by service consumers Changes in environmental quality 

Costs borne externally by other individuals Changes in resource use 

(outside the service or consumers) Changes in quality of life 











Source: Drummond et al. (9). 


Methods for identifying and measuring benefits are discussed fur- 
ther below in the sections on cost—effectiveness analysis and cost— 
benefit analysis. 


4. Dealing with uncertainty and time preference 

Whatever the method of evaluation used to economically appraise 
an environmental health service, the results still depend on various 
estimates. This is especially true when an economic evaluation is 
performed prior to conducting programme activities — for example 
when comparing the costs or economic effects of alternatives. In 
such cases, when data may not be available, gaps in knowledge need 
to be filled with default assumptions or through the use of models. 


Extensive reliance on default assumptions may prompt a sensi- 
tivity analysis, which determines the extent to which assumptions 
have influenced the results of the analysis. Sensitivity analyses also 
help in assessing which elements of an intervention (or a project) 
are most important to the final result. For example, a health service 
may display some astounding results, but these may be caused by 
factors not included in their tasks. Then, a sensitivity analysis can 
help uncover the actual links between inputs and outputs. It may 
indicate which variables in an analysis are most important, and which 
estimations affect the final results the most. 
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Another way to address the uncertainties involved in an economic 
evaluation is to organize meetings with key stakeholders (including 
experts and representatives of the public) to agree on the estimates 
and assumptions to be used in the evaluation process. When changes 
are recommended, the economic models must subsequently be al- 
tered to reflect the new estimates. 


Use of the discounting ratio is common in economic calcula- 
tions. Where costs and benefits are an issue, society generally pre- 
fers the present over the future. Benefits are desired in the present, 
while it is desirable that costs be postponed for the future. Thus, in 
long-term, large-scale programmes, especially, future costs and ben- 
efits are discounted (given less weight) to account for the time pref- 
erence. The discounting procedure itself is relatively simple (see 
Box 13). Choosing the discounting ratio, however, is an important 
decision that affects the outcome of the calculations. Choosing the 
ratio often requires a good deal of discussion, because preferences 
may differ widely across groups. For example, a funding organiza- 
tion and the individuals who are affected by the interventions may 
prefer a different ratio. When disputes occur, a sensitivity analysis 
can help determine whether (and to what extent) the ratio affects 
the outcome of the evaluation. Whatever ratio 1s chosen, evalua- 
tors should be prepared to defend their choice (for example, it 
was based on a national standard, stakeholder consensus or his- 
torical precedent). 


5. Reaching a conclusion 

The final step is to apply a decision rule to reach a conclusion, by 
asking: Does (or will) the service work efficiently? There are vari- 
ous decision rules for each of the economic evaluation tools. For 
example, the benefit-to-cost ratio can be calculated by dividing the 
value of the benefits by the value of the costs. If the result is greater 
than 1, the investment is (economically) worthwhile. But this is just 
one example; in the discussion of four economic evaluation tools 
that follows, more attention is given to other decision rules. 


Four economic evaluation tools 

Table 7 identifies four economic evaluation tools: cost—effectiveness 
analysis, cost-benefit analysis, cost-minimization analysis and cost— 
utility analysis. All four will be discussed in further detail. Particular 
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Box 13. Discounting procedure 





The discounting procedure tries to identify the present value (PV) of costs or 
benefits. This is done by using the following formula: 


C 
(14+ /)? 


PV = 





where C is the amount of money to be spent or gained in the future, / is the 
discount rate, and nis the number of years before the money is spent or the 
economic benefit realized. 


For example, an investment in a small environmental health service is expected 
to have a benefit of US $100 000 over 30 years, and a cost of US $20 000 in 
10 years. If the discount rate is assumed to be 5%, the present value (costs) is 
US $12 278 and the present value (benefit) is US $23 138. 


PV = US $20 000 /(1 +0.05)" 


cost 


PV, = US $100 000 / (1 + 0.05)*° 


benefit 


= US $12 278 

= US $23 138 

A benefit-to-cost ratio can be calculated by dividing the present value of costs 
into the present value of benefits. 


PV enen’ PVs = (US $23 138) / (US $12 278) = 1.88 
If we choose to use the decision rule of recommending investments with benefit-to- 
cost ratios greater than 1, this investment would be recommended for approval 
because the benefit-to-cost ratio is 1.88. 


We can perform a sensitivity analysis by changing the discount rate and view- 
ing the results. For example, we make calculations that use discount rates of 
8% and 3%. 


8% discount rate: 


PV ost = US $20 000 /(1+ 0.08) = US $9264 
PV sneri = US $100 000 / (1 + 0.08)%° = US $9338 
penet! P Voost = (US $9338) / (US $9264) = 1.07 


3% discount rate: 

= US $20 000 /(1+0.03)'° = US $14 882 
= US $100 000 / (1 + 0.03)% = US $41 199 
= (US $41 199) / (US $14 882) = 2.77 


benefit 


IPV 


benefit cost 


Notice how the use of a discount rate of 8% reduces the benefit-to-cost ratio to 
just above 1, while the use of a 3% discount rate pushes the ratio to nearly 3. 
These calculations are very sensitive to the discount rate chosen, and it should, 
therefore, be selected carefully. 








emphasis is placed on cost—effectiveness analysis and cost-benefit 
analysis, as these are the tools used most often in (environmental) 
health service evaluation. Both tools compare inputs to outputs, but 
there are some clear differences between them. 
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Cost—effectiveness analysis 

Cost—effectiveness analysis tries to identify the intervention that mini- 
mizes the inputs or costs, given a specific objective. Inputs or costs 
(on one side) are compared to outputs or health effects (on the other 
side). When using such an analysis, aspects that may be of interest 
to the evaluator may be difficult to measure or simply unavailable. 
In such cases, effects are often expressed as intermediate outputs. 
For example, instead of measuring the exact change in health status 
that results from air pollution control, which is very hard to quan- 
tify, it might be easier to measure changes in environmental param- 
eters (for example, the concentration of sulfur dioxide) as an inter- 
mediate effect. 


When identifying costs and effects of environmental health 
services, an evaluator is faced with a number of difficult ques- 
tions that should be taken into account during and after the evalu- 
ation. 


e Are there any other factors besides the environmental health 
service that affect the health, wellbeing, or productivity of peo- 
ple or groups? 

e Are there any negative costs, such as health expenditures saved 
because of a specific intervention, that can be considered as 
benefits of the service? 

e Are there additional costs and benefits generated by ex- 
tended life (measured in life-years) that should be taken 
into account? 


Cost—effectiveness analysis is especially useful in comparing dif- 
ferent courses of action within one project. Because benefits are 
defined through an objective (such as life-years saved), alternative 
courses of action must aim at the same objective. For example, an 
alternative that strives to maximize the number of life-years saved 
should not be compared to an alternative that prevents occupational 
hazards. An important assumption made in cost—effectiveness analy- 
sis is that the objective (change in health status or in environmental 
parameters) 1s worth achieving. 


In general, we can distinguish two ways in which cost—effectiveness 
can be used. 
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1. Select an alternative course of action, given a specific objec- 
tive. This strategy is also called cost-minimization, and is dis- 
cussed further. 


2. Maximize a particular objective, given a specific budget. For 
example, cost—effectiveness analysis could help answer the 
question: Which of the following activities will result in the 
highest reduction in cases of food poisoning for, say, an 
amount of US $200 000? 


Furthermore, cost—effectiveness analysis is especially useful in 
cases where: 


e different alternatives are aimed at one objective; 

e the objective is defined clearly and given from the outset of 
the evaluation; and 

e reaching the objective defined has little effect other than di- 
rect effects on the individual or the environment, such as on 
specific socioeconomic developments or activities by other 
organizations. 


Cost—effectiveness analysis is limited for several reasons. First, 
the objective chosen should be reliable and valid (see the section on 
objectives for indicators (page 55) for definitions of these terms) 
and should allow for a fair comparison between alternatives. Sec- 
ond, benefits tend to be defined narrowly, as health effects or inter- 
mediate outputs, which means that wider effects of the environmen- 
tal health service may not be taken into account. This may include 
such issues as the effects on productivity of individuals or the equi- 
table distribution of benefits across the population. Finally, cost— 
effectiveness analysis does not allow a comparison between differ- 
ent objectives. Thus, it cannot be used to decide which objectives 
are worth achieving. 


The most important and difficult step in performing a cost— 
effectiveness analysis is not so much the identification of alterna- 
tives or the measurement of costs, but the measurement or estima- 
tion of effects or benefits. How, for example, do we measure im- 
proved quality of life, changes in health status, or changes in envi- 
ronmental conditions? This is an important question and answering 
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it forms the key to a sound economic evaluation. The indicators 
discussed in Chapter 3 are the instruments used to determine 
changes in health status and environmental conditions. In gen- 
eral, health status indicators (see also Box 14) are most useful in 
cost—effectiveness analysis because the primary objective of many 
environmental health services is usually to promote or prevent an 
effect in humans. In many cases, however, environmental, process, 
socioeconomic and urban indicators are also used. 


Cost—effectiveness analysis generally relies on a single decision 
rule: costs divided by effects. The result is expressed as the cost in 
monetary units per effect unit. Examples of cost—effectiveness analy- 
ses are presented in Boxes 15 and 16. 


It is one thing to measure health or environmental changes, but it 


is something else to value and appraise them in monetary terms. 
Cost—effectiveness analysis does not assign monetary values to the 


Box 14. Measuring health status 





In evaluating cost—effectiveness a number of tools can be used to quantify 
changes in health status. 


Health indices 

Health indices are derived from validating and ranking (for example, between 0 
and 100) health statuses. Thus, various health statuses are given different 
values, which can be used in cost—effectiveness analysis. The indices may 
involve different dimensions of health. (See, for example, Kind et al. (70).) 


Health profiles 

These profiles take into account the multidimensional nature of health and, 
thus, provide a comprehensive picture of health status. This is done by using 
questionnaires on a number of topics, such as mobility, emotional reaction and 
pain. The use of this type of instrument is sometimes limited in cost—effectiveness 
analysis. (See, for example, McEwan (11).) 


Disability scales 

These scales are used to assess the extent to which individuals are able to 
perform basic daily activities. They usually reflect only one dimension of health. 
Most of these scales are not directly applicable to environmental health ser- 
vices, but there are some scales that focus on psychological aspects of health 
and wellbeing. These might be useful. (See, for example, World Health Organi- 
zation (12).) 


Information on health status is also available through the sources mentioned in 
the section on health indicators (page 69). 
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Box 15. Cost—effectiveness of noise complaint handling 





Three different scenarios that use cost—effectiveness analysis are presented, 
based on a hypothetical programme (A) that handles noise complaints on a 
yearly basis. This programme costs US $30 000 and the final result is an annual 
number of 4900 complaints handled. 


1. Cost-—effectiveness of programme A can be calculated as: 
costs / total number of complaints = cost per complaint handled; or 
US $30 000 / 4900 = US $6.12 per complaint handled 


2. Sometimes a variation of this calculation, the excess cost per complaint, is 
useful. Excess costs are calculated by subtracting any known monetary 
benefits from the costs, thus giving a more complete picture of the situation. 
For example, suppose that 290 of the 4900 complaints are considered 
justified by programme A’s service managers. Furthermore, assume that 
each of these justified complaints is valued (before carrying out the evalu- 
ation) at about US $100 each. The excess cost can be calculated as fol- 
lows: 


US $30 000 — (290 Xx 100) = US $1000 excess cost 


Using the excess cost as the numerator in the first equation yields the 
excess cost per complaint: 


US $1000 / 4900 = US $0.20 excess cost per complaint 


3. Itis also possible to compare programme A to other programmes. Take, for 
example, programme B, which operated a few years ago on a smaller 
scale. Programme B handled 500 complaints during the year it operated at 
a cost of US $2500. Fifty complaints were considered justified, and the 
value of the justified complaints is the same as for programme A. This 
means that the excess cost was: 


US $2500 — (US $50 X 100) = — US $2500 


In other words, when only the excess costs are considered, programme B 
was more cost-effective than programme A. The reason for this can, in part, 
be found in the “hit ratio” of each programme. Of all the complaints handled 
by programme A, only 6% (290 of 4900) are justified, whereas in pro- 
gramme B, 10% (50 of 500) are justified. 


This example shows that evaluators should carefully consider the values used 
as inputs in cost—effectiveness equations, even though the calculations them- 
selves may be fairly simple. 








effects, but instead incorporates the effect itself in the equation (that 
is, cost per complaint, as in Box 15, or cost per life-year saved, as in 
Box 16). Cost—utility analysis (discussed below in further detail) 
deals with the issues of appraisal. 
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Box 16. Cost—effectiveness of life-saving interventions 





In 1995, a team of seven American researchers published the results of a 
comprehensive study on the cost-—effectiveness of life-saving interventions. 
These interventions included behavioural and technological changes that re- 
duced the risk of premature death. Cost—effectiveness was defined as the net 
resource cost of an intervention per year of life saved. In total, 587 interventions 
were studied. Some examples of the results are: 


Life-saving Cost per life-year 
intervention saved 
Mandatory seat belt use law US $69 

Airbags in cars US $120 000 

Arsenic emission standard at copper smelters US $36 000 

Arsenic emission control at glass manufacturing plants US $51 000 000 

Chlorination of drinking-water US $3100 

Ban on amitraz pesticide on pears US $350 000 





Source: Tengs et al. (13). 


Cost—benefit analysis 

While cost—effectiveness analysis presents effects in terms of ob- 
jectives, cost-benefit analysis focuses strictly on both costs and ben- 
efits in monetary terms. Cost—benefit analysis aims at developing 
measures that assist in decision-making, in order to make choices 
among different intervention alternatives. A crucial distinction be- 
tween cost-benefit analysis and cost—effectiveness analysis is that 
the alternatives evaluated in a cost-benefit analysis need not be 
from the same project or objective, because the method translates 
all costs and benefits into simple monetary terms. 


Cost—benefit analysis is, therefore, especially useful in compar- 
ing different actions, which may have differing and multiple out- 
puts (or outcomes). Furthermore, it is an important tool for assess- 
ing whether the objectives chosen are worth achieving from a fi- 
nancial point of view (which is not possible directly using cost— 
effectiveness analysis). Finally, cost-benefit analysis can be useful 
when the evaluation needs to account for widespread effects of en- 
vironmental health service activity. These widespread effects (such 
as those on productivity) are especially relevant in large-scale, long- 
term interventions, such as securing safe and sufficient water sup- 
plies or industrial pollution-control measures. 
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Although cost-benefit analysis may seem to offer solutions to 
many problems, it is far from a perfect tool. The main problem is the 
valuation of benefits in monetary terms. This is not always possible, 
and it relies on a number of assumptions. More attention to the issue 
of measuring benefits is given later. There is also a debate on whether 
benefits should always be translated into monetary terms. Numer- 
ous situations exist where monetary valuation seems inappropriate — 
or highly impractical. How do we, for example, appraise in dollars 
the value of noise complaints, or the ability of a child to live without 
suffering the effects of asthma? 


Cost—benefit analysis is based on estimating the net economic 
flux (change). This quantity is the difference between economic ben- 
efits and costs generated by a service. While measurement of costs 
is generally easy, measurement of benefits can be difficult. Over the 
years many approaches have been developed to make it possible to 
put monetary values on such entities as environment and health. We 
will discuss two of them: the human-—capital approach and the con- 
tingent valuation methods. 


Human-—capital approach. The human—capital approach puts a value 
on the benefits of a particular service by estimating the additional pro- 
ductivity of the people (during their productive years) affected by the 
service. It requires a knowledge of the effects of environmental diseases 
on absenteeism (lost work days) and on the quality of work, and as- 
sumes that environmental conditions affect health status. Health status 
subsequently determines the quality of work performed, the productiv- 
ity of labour and the availability of labour for work (Fig. 15). Improve- 
ments in health status can thus be appraised in terms of increased and 
improved production and then translated into monetary values (/4). 


To evaluate the effects of productivity loss in physical terms, 
evaluators must start with the existing labour market and calculate 
employees’ productivity over the duration of their careers. Produc- 
ing outcomes based only on productivity have a limited scope be- 
cause certain groups (such as the long-term unemployed, the elderly 
and volunteers) may be systematically overlooked. 


Contingent valuation methods. Contingent valuation simulates a 
market price for goods that are not traded on the open market. Surveys 
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Fig. 15. Human-capital approach 


Environmental Health Productivity 
conditions status Quality of work 
Availability for work 


Source: Johannesson (14). 











and interviews are generally used to estimate “willingness to pay,” a 
measure of how much consumers are willing to pay to profit from 
an environmental health service. Contingent valuation can also be 
used to determine “willingness to accept’, the financial compensa- 
tion customers require to accept negative consequences. (Survey 
and interview techniques are discussed in greater detail on page 110). 


Contingent valuation methods have been used for both health 
economics and environmental economics. Methods to assess will- 
ingness to pay or willingness to accept are considered especially 
useful in the following applications: 


e environmental noise and stench 
e visibility benefits 

e recreation 

e physical planning 

e water quality 

e health benefits. 


Studies using contingent valuation methods are generally applied 
to small populations and are mostly related to local or regional envi- 
ronmental problems. An example of willingness to pay for safe food 
is presented in Box 17. 


Contingent valuation has problems that limit its usefulness. Tech- 
niques used to gather data on individual preferences are sometimes 
subject to several forms of bias: responses may be influenced by the 
wording of the questions, scenarios may be interpreted differently 
and respondents may be intentionally dishonest. The information 
supplied to individuals is never perfect, which further complicates 
estimates of willingness to pay. Also, respondents generally do not 
have experience in valuing the items of interest, and they may pro- 
vide implausible answers (for example, willingness to pay thousands 
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Box 17. Willingness to pay for safe food 





In Scotland, infected poultry accounts for about 42% of all recorded outbreaks 
of foodborne illness. One measure to secure safe poultry is to expose it to 
doses of ionizing radiation, a process called irradiation. Though the process is 
considered safe by WHO and the United Kingdom Government's Food Advi- 
sory Committee, both consumers and food retailers are not convinced of its 
safety. To measure the actual benefits of preventing poultry-borne risks, re- 
searchers at Aberdeen University completed a feasibility study to estimate the 
willingness to pay for such benefits. This was done by asking a small group of 
people how much (as a percentage of the amount already spent for poultry and 
meat on a weekly basis) extra they would be willing to pay for: 


1. poultry meat treated by irradiation 

2. poultry meat treated by a hypothetical preventive measure 

3. poultry meat not treated by irradiation. 
Individuals were provided with a questionnaire, including such questions as: 
What is the maximum additional amount you would be willing to pay, as a 
percentage of the amount you already spend each week, for poultry meat that 
had been irradiated and carried no risk of illness? 
The results, based on the questionnaire, were as follows. 


Mean willingness to pay for: 


irradiated meat 10.8% more 
meat treated by a hypothetical preventive measure 9.1% more 
meat not irradiated 3.6% more 


An aggregate (total) measure of the willingness to pay to avoid pain and suffer- 
ing associated with poultry-borne illnesses was estimated to be £6.5 million 
(based on the results of the study and such figures as the total United Kingdom 
spending on poultry meat). This figure far exceeds the total cost of irradiation, 
estimated at £2.5 million. 


Though this was a feasibility study carried out with a small population, it shows 
the possibilities of willingness to pay as an approach to estimate the individual 
benefits of preventive food safety measures. 








Source: Donaldson et al. (15). 


of dollars even when they are not likely to be able to afford such a 
sum). Another difficulty is that results may be highly specific to 
individuals, preventing generalization of results to wider audiences. 
In addition, because scenarios are hypothetical (no real money 
changes hands), there may be wide discrepancies between what re- 
spondents say they would do, and what they would actually do in a 
real life situation. Finally, the approach tends to favour the prefer- 
ences of individuals or groups that have more monetary power, rais- 
ing issues of equity for other segments of the population. 
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Box 18. Example of cost-benefit comparison 





The Ministry of Health demands an evaluation of the costs and benefits of 
health service activities. An external evaluation office focuses on environmen- 
tal health promotion and estimates the costs and benefits of that element in 
three different local services. The costs and benefits of three service elements 
are compared, and both the ratios and differences calculated in arbitrary mon- 


etary units. 
Service Costs Benefits Benefit-to- Difference (net 
cost ratio present value) 
C B B/C B-C 
| 10 000 45 000 4.5 35 000 
j 40 000 120 000 3.0 80 000 
III 5 000 30 000 6.0 25 000 


Based on the benefit-to-cost ratio, one might conclude that service III is the 
most efficient (the value 6.0 is higher than 3.0 or 4.5). The ratio, however, does 
not indicate the size of the programme; therefore, the cost-benefit difference, 
or net present value, is often chosen as the decision rule. In this case, service II 
would most likely be labelled as most efficient. Since these calculations explore 
different aspects of service costs and benefits, it is recommended that they be 
used together to assess services. 





Two decision rules are available for comparing alternatives using 
cost-benefit analysis. The first is to calculate the benefit-to-cost ra- 
tio, where benefits are divided by costs. If the result is more than 
one, the activity is (potentially) determined to be worth while. A 
second option is to calculate the net present value by subtracting the 
costs from the benefits. This is often the preferred rule. If the result 
is larger than zero, the action is judged to be worth considering. 
Box 18 illustrates a cost-benefit comparison using both these rules. 


Cost-utility analysis 

Cost—utility analysis is similar to cost—effectiveness analysis in that 
both methods calculate the costs involved in implementing different 
alternatives. Cost—utility analysis, however, allows individuals to place 
a specific monetary value on the benefits (that is, changes in health 
status or environmental conditions measured), whereas cost— 
effectiveness analysis does not. Cost-utility analysis also allows the 
evaluator to take quality of life into account, while cost—effectiveness 
analysis does not. 


Utility refers to the relative usefulness that an individual assigns 
to a specific health status or environmental condition. The utility 
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value is gathered by using questionnaires and survey techniques 
and, ultimately, results in an adjusted unit of benefit. For example, 
the number of life-years saved can be valued and adjusted for the 
quality of those years, resulting in quality-adjusted life-years (QALYs) 
(16). QALYs thus combine quality of life information with mortality 
data to measure treatment or policy outcomes. The number of life- 
years saved can also be adjusted for disability, resulting in disabil- 
ity-adjusted life-years (DALYs) (17). DALYs measure the burden of 
disease, reflecting the total level of healthy life lost to all causes, 
including premature mortality and physical or mental disability. 
Though this form of economic evaluation has some limitations (such 
as forms of bias in gathering data on utility), it is a common ap- 
proach used in health economics. 


Cost-minimization analysis 

This simple tool aims to minimize the costs of a given course of 
action, under the assumption that the final output is the same in 
every alternative. This assumption should not be made without suf- 
ficient evidence that the alternative courses of action have similar 
outcomes. Cost—minimization analysis is often combined with stud- 
ies that analyse programme outputs (or effects). The decision rule 
for cost-minimization analysis is relatively straightforward: pick the 
least expensive alternative. Cost—minimization analysis is a tech- 
nique for saving money, but it is less relevant than the others for 
evaluating environmental health services and is, therefore, not dis- 
cussed further here. 


Table 8 summarizes the four economic evaluation tools. 


Precautions for use of economic evaluation tools 
Several final points of advice and precaution are warranted on the 
subject of economic evaluation tools. 


Intersectoral evaluations. Environmental health is a multi- 
sectoral endeavour. Environmental quality is not influenced by 
environmental policy alone. Similarly, health is not affected by 
environmental quality alone. This calls for an approach to eco- 
nomic evaluation that makes optimal use of the methods de- 
scribed. It is highly recommended that evaluators combine dif- 
ferent tools. 


Table 8. Summary of economic evaluation tools 








Tool Measure of costs Measure of consequences Activities tool considered References 

most useful for: 

Cost- Costs expressed Outputs measured in a single, »* comparing different courses of action to Drummond et al. (9) 
effectiveness in monetary units common effect reach a single objective Tengs et al. (13) 
analysis Outputs expressed as physical e maximizing an objective, given a specific Graham & Vaupel (18) 

units, e.g. life years saved budget 

and cases prevented e cases where the objective has little effect 
outside the direct effect on the individual 
or the environment 

Cost- Costs expressed Outputs measured by different * comparing different actions, with differing Drummond et al. (9) 
benefit in monetary units effects (single or multiple) that and multiple outputs or outcomes Thompson (79) 
analysis may differ from one alternative +e assessing whether objectives chosen are Donaldson (20) 

to another worth achieving 
Outputs expressed in * accounting for more widespread effects of 
monetary units environmental health service activity 

Cost- Costs expressed Outputs measured by different * comparing effects of a different nature Drummond et al. (9) 
utility in monetary units effects (single or multiple) that e accounting for issues of quality Richardson (27) 
analysis may differ from one alternative 

to another 
Outputs expressed as healthy 
days, QALYs or DALYs 

Cost- Costs expressed Outputs assumed to be similar * selecting an alternative course of action, Drummond et al. (9) 
minimization in monetary units No outputs valued given a specific objective 
analysis 


NOILVN TWAS AO SLNANNYLSNI 


LOL 


108 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5 


For whom? An economic evaluation, like any other, is done for a 
specific audience with specific goals in mind. This affects both the 
evaluation design and the data needs. 


Economic evaluation is not perfect. Economic evaluation tools 
have several limitations. Evaluators are faced with a number of un- 
certainties and, therefore, rely on a number of assumptions. Further- 
more, costs and benefits are not the only factors of importance in 
evaluation and decision-making. Finally, economic criteria are, gen- 
erally, not the only factors in the decision-making process; they must 
be balanced with many additional considerations, such as equity 
and risk. 


Equity issues. When dealing with the costs and benefits of a spe- 
cific activity, evaluators should not neglect the issue of equity. Eco- 
nomic evaluation neither always indicates who benefits from an al- 
ternative action nor accounts for the distribution of benefits across 
the community. For example, a water treatment programme at the 
local level may be very cost-effective: at a low cost, a large popula- 
tion has access to supplies of safe drinking-water. But it may mean 
that at the regional level many other individuals lack this opportu- 
nity. In such a case, fairness may prevail over strictly economic con- 
siderations. 


Other economic instruments 

This section has primarily focused on economic analysis techniques 
such as cost-benefit analysis and cost—effectiveness analysis. Other 
economic instruments, such as tax incentives and pollution permits, 
have been used in the European Region and elsewhere for setting 
priorities and for fiscal reforms. Additional information on economic 
instruments can be found in a recent background paper (22), but it is 
not discussed in this book. 


Qualitative evaluation 

Qualitative research and evaluation methods allow evaluators to study 
a small number of specific issues in great depth and detail. They do 
not seek answers to such questions as: “How much is X?”, as in 
quantitative research — but rather “What is X?” and “Why does X 
occur?” Often qualitative and quantitative approaches are defined 
as opposites, but they should be seen as complementary rather than 
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conflicting approaches. Each has its own strengths and limitations, 
as expressed by Patton (4): 


The advantage of a quantitative approach is that it’s possible to measure the 
reactions of a great many people to a limited set of questions, thus facilitating 
comparison and statistical aggregation of the data. This gives a broad, 
generalizable set of findings presented succinctly and parsimoniously. By 
contrast, qualitative methods typically produce a wealth of detailed informa- 
tion about a much smaller number of people and cases. This increases under- 
standing of the cases and situations studied, but reduces generalizability. 


Quantitative research generally involves stating a hypothesis (usu- 
ally derived or deduced from an existing set of hypotheses or theo- 
ries), testing it (sometimes using experiments), and using deductive 
reasoning to come to a conclusion. Qualitative research by its very 
nature is different: it is more open and uses inductive reasoning to 
work from hypotheses to develop a set of assumptions and, ultimately, 
a theory. It is less limited in its initial phases than the preconceived and 
more regimented approach of experimental inquiry. Qualitative evalua- 
tion can tap into areas that quantitative methods cannot reach (23,24). 


In the sections that follow, the terms qualitative and quantitative 
are used as polar opposites, but the distinction between them is not 
so clear. Often the methods are used together to complement one 
another. An example of this can be seen in the economic evaluation 
techniques discussed earlier. Environmental economics is typically 
thought of as a quantitative enterprise: first, monetary values are 
placed on goods that are not traded on an open market and, then, 
mathematical equations are used to compare alternatives. Qualita- 
tive methods, however, are commonly used in such techniques as 
contingent valuation to assign monetary values to inputs and out- 
puts. Again, because they complement each other, qualitative and 
quantitative approaches are quite often used together. 


Overall strengths and weaknesses of qualitative evaluation 

The primary strength of qualitative evaluation is that substantial de- 
tail (usually written) about a given subject allows a user to make 
personal judgements about the subject. Validity and realism are strong 
features of the qualitative approach, because the objective is to as- 
sess specific circumstances rather than to make general statements 
about a broad situation. Another strong feature is that qualitative 
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evaluation methods allow flexibility, insight and the ability both to 
build on tacit knowledge (what is known but not stated) and to gen- 
erate facts that are not directly available through quantitative inquiry. 
Because qualitative analyses are based on real situations, they can 
increase the credibility and understanding of a particular perspec- 
tive. The open-ended nature of questions and inquiries allows an 
unbounded and, therefore, potentially innovative range of possibili- 
ties. Qualitative evaluation, therefore, may foster a dynamic exchange 
between the data collected and the resulting conclusions (processes 
that involve two distinct and sequential stages in quantitative inquiry). 


The extent to which qualitative evaluations can be used is limited 
by the degree to which data collected can be generalized to a large 
population, because these data are dependent on the context in which 
they are collected. Internal and external reliability are also compro- 
mised to a certain degree — that is, conclusions are not necessarily 
applicable in other circumstances. In addition, because qualitative 
analyses can vary with the particular individuals who collect, ana- 
lyse and present the data, they are limited by the skills and abilities 
of those individuals. For example, observations will vary among 
data collectors, in either participatory or interview settings. Unin- 
tentional biases can be introduced by personal situations (such as 
fatigue), evolution of the evaluator’s knowledge about the situation, 
and differences in training, skills and experiences. Also, because 
qualitative methods are more labour intensive, they may sometimes 
be a more expensive methodology. 


Qualitative evaluation tools 

Five basic types of qualitative tool are discussed in some detail in 
the sections that follow: surveys and interviews, focus groups, di- 
rect observation, review of written documents, and case studies. Con- 
sensus methods, such as the Delphi method and expert panels, are 
sometimes used in health service research (25) to facilitate agree- 
ment within a group on a particular topic. Since consensus methods 
are less likely to be useful in an environmental health service evalu- 
ation, they will not be discussed further. 


Surveys and interviews 
Surveys and interviews are popular techniques used to elicit infor- 
mation directly from people (in their own words) about perceptions, 
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experiences, opinions, feelings and knowledge of a specific situa- 
tion. Surveys and interviews are among the most basic forms of 
qualitative data collection. Two types of question can be asked in 
either an interview or survey format: open-ended questions and 
closed questions. Closed questions have a defined set of potential 
responses, while open-ended questions have no pre-established for- 
mat (or boundary) on responses. For example, any question that 
leads to a “yes” or a “no” answer is a closed question. 


Example 
Have you ever lodged a complaint with your local environmental 
health service? 


Another type of closed question asks respondents to choose from 
several alternative predefined options. Such answers allow responses 
to be analysed statistically, but are limited by the preconceived no- 
tions of the surveyor or interviewer. 


Example 
How satisfied are you with the training you have received for this 
position? 


— Notat all satisfied ___ Satisfied 
___ Somewhat satisfied ___ Very satisfied 


Another possibility is to allow respondents to choose between 
answering a question with a predetermined response, or writing or 
describing their thoughts in their own words. This type of question is 
more likely to provide new information to the qualitative evaluator 
than questions with simple “yes” or “no” answers; it also allows the 
surveyor to categorize several responses and still leave room for 
additional (unexpected) responses. This type of question is referred 
to as “mixed”. 


Example 
How did you hear about this environmental health clinic? 


___. Newspaper ___ Friend 
___ Radio —__ Doctor referral 
Other 
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Open-ended questions can be part of written surveys, and inter- 
views. Such questions allow respondents to answer in any way they 
see fit. While more difficult to analyse across many respondents, 
responses to open-ended questions can provide significant insight 
into issues that are seen by the respondent to be important. Open- 
ended questions are asked both in written (survey) and oral for- 
mats. 


Survey example 

The following space is provided for you to comment on the over- 
all delivery of waste management services in your county. Please 
feel free to voice other related concerns, or expand on earlier 
comments. 


Oral examples 

What would you tell a new employee about working at the 
ministry? 

Describe the most important factors in your decision process. 
What is your reaction to the new legislation? 


Interviews are usually classified in one of three ways: structured, 
semi-structured or in-depth. The structured interview is a personal 
discussion, following a planned, predetermined format. Typically the 
interviewer has a questionnaire with a series of closed and mixed 
questions. In a structured interview, each person would be asked the 
same questions in the same order, and variability across responses 
would be narrow compared to other interview techniques. Structured 
interviews differ from surveys in that during an interview, the inter- 
viewer captures information, whereas in a survey, the respondent 
writes his or her own responses. 


The semi-structured interview consists of a preplanned, predeter- 
mined set of mixed and open-ended questions. All people interviewed 
are asked the same questions, but not necessarily in the same order. 
Variability across responses would be larger, but each semi-structured 
interview is likely to have a similar structure. 


The in-depth interview is a detailed (sometimes lengthy) open- 
ended discussion. In this format, topics are often predetermined, but 
specific questions may or may not be developed before the interview. 
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Questions in this type of interview vary with the respondent’s expe- 
rience or expertise, and each question builds on the prior response. 
Variability across respondents is wide, and most in-depth interviews 
would have somewhat unique structures. 


Semi-structured and in-depth interviews allow personal perspec- 
tives to be collected, thereby providing a certain element of credibil- 
ity that a simple number or statistic is often unable to provide. Lis- 
tening to (or reading) examples of real situations in the words of the 
person who experienced them is extremely valuable, and may 
yield insights that surveys cannot provide. In addition, answers 
to open-ended questions can lead to hypotheses that can be tested 
on wider audiences, which is particularly helpful for exploratory 
research (26). 


Surveys are limited by the writing skills of respondents, and by 
the amount of detail a respondent can provide in the time allotted for 
a group of questions. Furthermore, it is difficult to go back to a 
survey respondent for more information, while it is possible to ask 
additional follow-up questions in an interview. Both surveys and 
interviews are limited in that they are open to different interpreta- 
tions and biases of interviewers and readers. Good training and ex- 
perience can reduce bias, but they are unlikely to eliminate it com- 
pletely. Box 19 applies each of these formats to examples within the 
context of environmental health service evaluation. 


Several techniques are important to consider when conducting 
interviews and surveys. 


e Asking good questions is an acquired skill; a variety of tech- 
niques can be employed, which may differ with each situation 
(such as individual interviews, informal conversations and 
group interviews). 


* Questions should be structured so that they are as neutral as 
possible. Questions should not lead respondents to answer in 
a particular way. 


* Questions should be focused and direct, and should clearly 
reflect the purpose at hand. 
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Box 19. Examples of types of interview 





Structured interviews 

A questionnaire could be used to assess public opinion on a range of local 
issues (such as air pollution, water treatment or soil clean-up) addressed by an 
environmental health service. The technique is useful: it can be applied to a 
relatively large group, and clear-cut answers are possible. Similarly, a written 
survey can be carried out to identify public concerns. Since it requires little 
manpower, a large group can be surveyed. 


Semi-structured interviews 

The semi-structured interview puts specific demands on researchers and evalu- 
ators, and usually only a limited number of interviews are possible. The tech- 
nique, however, provides more detail on people’s feelings and beliefs on certain 
issues. It could, for example, be used to analyse people’s satisfaction with the 
way a service works. In the same way, it could be used to assess opinions of 
service professionals — for example, on service organization and specific pri- 
orities set by the service. 


In-depth interviews 

This technique puts even more demands on researchers and limits the number 
of people who can be included. Because environmental health services perform 
tasks for a very diverse audience, this technique is applicable only in specific 
situations. For example, the evaluator could identify a small group of key in- 
formants. These might include spokespeople for relevant organizations, man- 
agers, service staff, trade unions and politicians. Presented with broadly worded, 
open-ended questions, such informants might provide valuable information on 
processes that relate to services. 








Role playing and simulation questions can also be used to 
make the interview more natural and less confrontational. 


Using illustrative examples in questions can build rapport be- 
tween the interviewer and the person being interviewed. Ex- 
amples can also communicate that an interviewer will not be 
judgemental about what is said because “it has all been heard 
before”. 


There is a trade-off between breadth and depth — that is, only 
a few topics can be covered in depth, or a greater number can 
be covered more broadly. 


Interviews usually involve several means of data collection: 
questions are asked, observations are made and documents 
are sometimes collected. It can be useful to videotape or 
audiotape interviews when such equipment is available. 
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e Many surveys contain a mixture of question types, so several 
types of technique can be used during analysis. If results cor- 
roborate one another, the findings are strengthened and con- 
clusions may be considered more valid and reliable. 


Focus groups 

A focus group is a type of group interview. Its most distinguishing 
feature is that the focus group uses interactions within a group in- 
stead of gathering answers to a specific set of questions. A focus 
group consists of a small number of individuals who embark on a 
group discussion facilitated by the evaluator or researcher. Discus- 
sion topics may vary, but in the past focus groups have been espe- 
cially useful in elucidating group beliefs, experiences and attitudes. 
For example, they have been used to assess professional satisfaction 
with the way a health service works, and also to identify the impact 
of health promotion messages. 


There are few strict rules for conducting a focus group study, but 
some attention should be given to group size and setting. Groups 
should not be too large (4-8 people) and the setting should be com- 
fortable, encouraging and familiar to the participants. As with most 
tools described in this volume, the use of focus groups should be 
combined with other means of gathering data. For instance, a ques- 
tionnaire is often used in conjunction with focus groups, and some- 
times it is useful to conduct one-on-one interviews either before or 
after group discussions (27). 


Direct observation 

Direct observation occurs when an evaluator observes activities in 
progress and records them. This tool is particularly useful for de- 
scribing people’s activities, their behaviour in certain situations, and 
the organizational context and process in which they work. The con- 
text of activities and situations to be evaluated is very important. In 
order to produce an evaluation document that allows the reader to 
make personal judgements about the situation, the evaluator may 
wish to describe the context as factually as possible, and in great 
detail. 


Objectivity exists in varying degrees, and situations may be in- 
terpreted differently by different people. Thus, it is often argued that 
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qualitative inquiry requires both adequate training and preparation. 
Not only must an observer be taught to see, hear and record infor- 
mation in a useful way, but he or she must actively prepare to do so 
(4). As an observer, a researcher (or evaluator) can play different 
roles, from a complete participant in a special activity or process (a 
technique seldom applied and often questioned for ethical reasons) 
to a completely uninvolved observer who embarks on studying ac- 
tivities as an experiment (28). The role an observer plays in a group 
greatly influences the reliability and validity of the results. Variation 
and bias in observers are clearly concerns, but they are generally 
outweighed by the depth and detail provided by directly observing 
or participating in activities. 


Direct observations are valuable for several different reasons 
(4,23). Participating in an activity, instead of strictly observing it, 
automatically improves one’s understanding of the situation. First- 
hand experience reduces the need to rely on preconceived notions. 
In addition, evaluators who participate in an interview are trained to 
pick out the things that other participants may overlook because they 
are accustomed to them. By participating in an activity an evaluator 
may discover sensitive issues that could be missed during a for- 
mal interview. Direct participation can also help the observer (or 
evaluator) understand and interpret data from interviews and re- 
views of additional documentation. 


Review of written documents 

Reviewing and summarizing programme documentation and litera- 
ture (often called document analysis) is another important tool used 
in qualitative research and evaluation. A technique called content 
analysis (see, for example, Weber (29)) assigns categories to rel- 
evant material in order to find patterns, linkages and common mean- 
ings. Documents of interest may include written material available 
within an environmental health service or other relevant organiza- 
tions, such as: 


e excerpts from books, quotations and entire passages from or- 
ganizational, clinical, and programme and service records; 


* memoranda, organizational charts, personal diaries and cor- 
respondence (including electronic mail); 
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e official publications (such as policy statements), publications 
in scientific magazines and service reports; and 


e posters and pamphlets, draft reports and written responses to 
questionnaires and surveys. 


Document analysis should not be confused with the review of 
scientific reference material. It is a means of gaining insight into the 
processes and structures within environmental health services by 
reviewing material that may be unofficial or unpublished, but that 
contains information highly relevant to the efficiency, effectiveness 
and/or equity of a service. Such files can be a rich source of back- 
ground and process information on an activity, as well as informa- 
tion on how decisions were made. 


In most cases, a review of documentation alone is insufficient 
for understanding the actual events. It is therefore often combined 
with other tools, such as interviews. The study of written or printed 
material can help focus interview questions and identify potentially 
new hypotheses. 


Document analysis, though highly useful, is not without prob- 
lems. In today’s world of computers, electronic mail and extensive 
paper trails, background documentation may be voluminous and, 
therefore, it may be difficult to know where to start. Also, knowing 
how to distinguish between important data and useless data is a skill 
that must be acquired through practice. Furthermore, documents tell 
only a part of the story and should not be substituted for information 
obtained from the direct personal contact of interviews and obser- 
vations. 


Case studies 

Another tool of particular importance to evaluation is the case study 
(30-32). Case studies analyse a limited set of cases (or situations) 
where the researcher expects to find a relevant phenomenon or where 
a certain hypothesis can be tested. For example, a national evalua- 
tion of environmental health services may be too costly to include 
all services at the outset. By identifying a limited number of cases, 
the most important strengths and weaknesses can be identified. These 
can then be used to further focus the broader evaluation. 
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Case studies are especially helpful in studying complex situa- 
tions that involve many factors, such as a large number of interested 
parties. This, for example, is the case with implementation of poli- 
cies. Studying a distinct number of cases can answer such important 
questions as: What happened in the course of the implementation? 
Why did the implementation process fail in this instance? Which 
factors contributed to success — or failure? Ultimately, case studies 
provide clues to why things might have happened in a specific situ- 
ation. Though the approach is limited, in the sense that its results are 
usually not generalizable on a large scale, it is often used as a some- 
what straightforward means of testing implementation processes and 
is valuable for gaining insight into specific circumstances. 


Another example of the use of this particular tool is provided in 
Box 20. 


By itself, the case study approach is neither qualitative nor quan- 
titative. It is, however, a unique way to look at individual situations 
so that information is gained and knowledge advanced. Thus, case 
studies often combine a mixture of interviews, surveys and focus 
groups with participatory, economic and statistical techniques. The 
case study approach is used in this volume to provide detailed ex- 
amples of environmental health service evaluations in the European 
Region, and to explore how they might be improved (see Chap- 
ter 6). The case studies employ several evaluation techniques em- 
phasized in this chapter. 


The use of qualitative evaluation tools 

The qualitative approaches described in this chapter have been used 
in a wide variety of settings, involving such subjects as the beliefs 
and perceptions of people, the organization of health services, and 
professional development and education. When looking at the envi- 
ronment and health, qualitative analysis can be applied to many of 
the types of evaluation discussed. Process and implementation evalu- 
ations, circumstances with individualized outcomes, and the evalu- 
ation of prevention activities are of particular interest. 


The evaluation of process and programme implementation can 
be enriched greatly by qualitative data. Quantitative analysis some- 
times provides only a yes or no answer (for example, yes the 
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Box 20. Implementation of Healthy Cities initiatives: 
three case studies 





Three Healthy Cities in the Montreal area of Canada were studied as cases of 
implementing initiatives for promoting health. The researchers were especially 
interested in evaluating the implementation of: 


e the sharing of common objectives among different interested parties; 
e intersectoral action; and 
e community participation. 


The evaluation included a strongly interactive component. Evaluators were in 
close contact with those being evaluated through a steering committee and by 
publishing a newsletter regularly. The interactive approach was used alongside 
a specific framework for performing the case studies, including the analysis of 
the following factors. 


e External environment. To what extent do things such as laws and overall 
political and economic context influence the implementation? 


e Internal environment. What features of the city or neighbourhood influ- 
ence the implementation process? 


e Coherence of objectives. Are common objectives officially shared be- 
tween different actors? 


e Results of implementing initiatives. What is the level of satisfaction 
among actors? How are changes perceived? 


This approach created awareness among actors of the three initiatives and 
resulted in useful clues as to how the implementation processes could be 
improved. 





Source: Ouellet (33). 


process was carried out or no the implementation was not timely). 
Qualitative analysis can help answer how and, especially, why cer- 
tain things did or did not occur. Many believe that in order to study 
or evaluate a process or implementation it should be experienced. 
Inductive reasoning in qualitative research can provide this per- 
spective. 


Qualitative approaches are also useful for evaluating properties 
that are expected to produce variable results in different individuals. 
For example, in a training exercise, individuals may begin with vastly 
differing skills, so the value added to their expertise will vary among 
them. The best way to understand the effect of the training thus might 
be to select a few individuals and complete detailed, yet manage- 
able, case studies of outcomes. Case studies are also appropriate for 
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innovative interventions and programmes — that is, when outcomes 
are very unpredictable. 


Prevention programmes are also suitable for qualitative evalua- 
tion techniques. As mentioned in Chapter 2, the long time-frames 
required for some environmental health outcomes, the lack of clear 
causal links between prevention programmes and environmental 
health outcomes, and the ethical implications of research on hu- 
mans all reduce the utility and appropriateness of long-term hypo- 
thesis testing on human trial and control groups. Furthermore, many 
programmes aimed at preventing environmental health problems can 
be linked to behavioural changes; qualitative techniques are par- 
ticularly suited to gathering details about how people’s attitudes and 
behavioural patterns change during an intervention. 


Precautions about qualitative evaluation tools 

Qualitative approaches rely on a certain level of flexibility. This ele- 
ment is sometimes difficult to manage. Practical exploratory tech- 
niques require the evaluator to have tolerance for ambiguity and 
uncertainty in the information gathered. Planners and evaluators must 
have faith that whatever conclusions are reached, there will be an 
inherent value in using qualitative methods to conduct the evalua- 
tion. Designers, however, can never know with certainty what the 
results will be; they can only look at results from similar studies and 
evaluations and decide what seems plausible. 


Qualitative techniques must be flexible, but they must also be 
rigorous and systematic. Developing and following a detailed evalu- 
ation plan helps to ensure the credibility and validity of the process 
and information. Other strategies to strengthen credibility and valid- 
ity include ensuring that representative samples are taken (when se- 
lecting individuals as key informants, for interviews and focus 
groups), recording what is observed as factually as possible, and 
representing the information found in a structured manner. 


DISCUSSION AND RECOMMENDATIONS 


The sections in this chapter provide the reader with an introduc- 
tion to several useful techniques for designing and implementing 
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environmental health service evaluations. Whatever design is cho- 
sen, evaluators should strive to ensure that their work represents 
reality as closely as possible, that the indicators measure as pre- 
cisely as possible the issue being studied, and that the evaluation 
will be as broadly applicable to other situations as possible. These 
three objectives — realism, precision and generalizability — can never 
be maximized simultaneously, and trade-offs among them are in- 
evitable (34). Finding the proper balance is more likely when more 
than a single type of indicator is collected, and when more than a 
single method of data collection or analysis is used. 


Assuring validity and reliability of results should be a fundamen- 
tal goal of any evaluation or research process. One way to assure 
scientific credibility during the evaluation process is to use a popu- 
lar technique called triangulation (35), which is based on taking more 
than a single perspective to produce a wholly accurate picture of a 
given situation. Thus, a strategy that presents more than one per- 
spective is desirable. Triangulation can be applied to methods, evalu- 
ators, data types and even theories. For example, an optimal mix of 
methods might include qualitative, experimental and statistical analy- 
ses, and would reflect the nature of the evaluation and situation at 
hand. 


This chapter has also pointed out that the tools applicable to en- 
vironmental health service evaluations have come from several dif- 
ferent scientific disciplines. This comes as no surprise, because an 
evaluation is a multidisciplinary effort. The social sciences (such as 
political science, anthropology, psychology and economics) have 
made important theoretical contributions, and statistics, biometrics, 
epidemiology and psychometrics are central to measurement and 
analysis. This rich mixture of disciplines suggests that environmen- 
tal health services should be evaluated by multidisciplinary teams. 
Evaluators may wish to introduce concepts or methods from any of 
the disciplines mentioned here, as well as from those not mentioned. 
Under any given circumstance, the disciplines to be invoked will 
depend on the knowledge available, the purpose of the evaluation 
and the questions at hand. 


Evaluators should also keep in mind that the multifaceted nature 
of environmental health services requires different strategies for 
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evaluating different situations. Because of their training, individuals 
in a certain discipline may have a limited view of what an “evalua- 
tion” should be. Before embarking on any specific evaluation, the 
strengths and weaknesses of the techniques should be carefully con- 
sidered in the context of the particular situation. 


Box 21 summarizes the key recommendations of this chapter. 


Box 21. Key recommendations from Chapter 4 





Recognize that all tools are not applicable to all evaluation settings and ques- 
tions 


Review the strengths and weaknesses of specific tools for gathering data 
before using them, and pick the tools most suitable to reach evaluation goals 


Consider using a mixture of evaluation tools simultaneously (triangulation) to 
strengthen the credibility of the analysis and the reliability of the results 


Recognize the complexity of environmental health service activities and the 
additional skills needed for evaluating such actions; aim for multidisciplinary 
evaluation teams 
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Making Evaluations Work 


In previous chapters, concepts, principles and tools of evalu- 
ation are discussed, along with various features of environ- 
mental health services. In this chapter, all these elements are 
brought together with the aim of producing evaluations that do 
not just sit on the shelf. Issues relating to dissemination of 
results, core factors affecting evaluation use, and advanta- 
geous evaluation qualities are presented. 


GENERAL ISSUES OF DISSEMINATION 


The fourth major step in an evaluation (see Fig. 4) is to disseminate 
results, conclusions and recommendations to the appropriate audi- 
ences. Much has been written on this subject (see, for example, 1—5), 
so only the most essential points are reviewed. 


First and most important, evaluation results should be com- 
municated clearly and succinctly, and should not employ too much 
scientific or organizational jargon. The structure and logic of re- 
ports and presentations must be comprehensible to the intended 
audiences. 


Making the best use of evaluations is an essential goal of this 
book. Because evaluation results are used in different ways by 


different groups of people, clearly establishing the purpose of an 
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evaluation helps to define the appropriate dissemination strategy. 
Specifically, evaluation results are mainly used to: 


e provide information and new knowledge; 

* stimulate new actions or adjustments within services, policies 
and plans; 

e stimulate participation, discussion and interaction with others; 

e justify expenditures; 

e account for actions taken; and 

e fulfil legislative requirements. 


In more general terms, evaluation results can be used in at least 
three ways: directly, conceptually or persuasively (6). Evaluations 
are said to have a direct (or instrumental) effect when the results are 
used directly to improve service delivery. In such cases, recommen- 
dations are taken directly into account during the decision-making 
process. For example, the evaluation of a chemical registration sys- 
tem may find that personnel spend too much time on data entry as a 
result of outdated software. If managers then decide to replace the 
outdated software with newer programs, the evaluation results will 
have been used directly. Evaluations are used in a conceptual way 
when the main principles or concepts outlined in an evaluation re- 
port are used in future planning but do not have direct effects. For 
example, an evaluation may find that a particular service is efficient 
but that little has been done to provide the public with equal access 
to services, even though equity may not have been a concern at the 
start of the evaluation. When such results are used to raise questions 
or stimulate new developments for future evaluations (for example, 
planning and allocation of funds), then they are being used in the 
conceptual sense. Finally, evaluation results are also used as tools of 
persuasion, as arguments for or against certain programmes, poli- 
cies or political viewpoints. For example, evaluations can be used 
persuasively to seek additional funding, redirect resources, reorgan- 
ize activities, and prioritize future actions. 


Since different groups use information for different purposes, 
evaluators should give explicit attention to the intended audience 
for their results. When evaluations are completed for internal needs, 
results are usually directed to managers, staff and other organi- 
zations responsible for overseeing the services. At other times, 


MAKING EVALUATIONS WORK 129 





evaluations are completed for broader purposes and the results are 
disseminated to a wide range of stakeholders. 


Goals and objectives for disseminating results should be clearly 
established during the planning phase of an evaluation. Once the 
purpose and audience have been identified, evaluators should consider 
when and how often results should be communicated. For both small- 
and large-scale evaluations, a final report is often written and released 
at the end of the process. But communication efforts do not always have 
to be formal. Communicating preliminary and interim results to inter- 
ested parties, even when done informally, can create a dynamic feed- 
back loop that allows the evaluators to make adjustments during the 
course of an evaluation, and can lead to organizational learning. 


The message also depends on the purpose and audience. Staff, 
management and funding organizations often desire an in-depth ac- 
count of the evaluation, but policy-makers and the general public 
may be less interested in technical details, preferring reports that 
focus only on results and conclusions. An evaluation team may 
choose to produce a technical report only for those interested, but 
may publish an executive summary for a broader audience. Techni- 
cal reports, generally, follow the evaluation plan outline and contain 
detailed accounts of the scope, purpose, methodology, analysis, results 
and conclusions for an evaluation (see, for example, Box 1). 


The format used to communicate results also deserves careful 
planning. A wide range of formats is available. The results can be 
published or announced in peer-reviewed journals, at conferences, 
in newspapers, through electronic media (the Internet, CD-rom) or 
in trade journals. Oral presentations can be given internally (for ex- 
ample, to staff and management) or externally (for example, to com- 
missioning organizations and the public). When results are provided 
electronically, attention to software compatibility is important. Early 
efforts to ensure that data can be imported to word processing, 
spreadsheet and mapping software can help guarantee that the re- 
sults of the evaluation are used by the intended audiences. 


Evaluators should direct the dissemination strategy by consider- 
ing who it is they want to reach, where those individuals and 
organizations are likely to get their information, and in what format 
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the information will be most useful. Results for practitioners, for 
instance, may contain methodological and conceptual considerations 
and be sent to peer-reviewed journals, while results intended for the 
general public may highlight only the main conclusions of the study 
and be published on the Internet and in newspapers. Sometimes, 
providing background or technical information may be required to 
help lay audiences understand the services being evaluated. 


A final issue is who should do the communicating. Some organi- 
zations or individuals may have differing levels of authority or cred- 
ibility with certain audiences. Identifying partners who can assist 
with the dissemination of results is often an effective component of 
a communication strategy. 


The case studies in Chapter 6 illustrate how evaluation results are 
communicated across the Region. In addition, the Regional Office 
produces a series of pamphlets that contain technical and practical 
advice on health and the environment. The pamphlets themselves 
are an example of how information is formatted with certain audi- 
ences in mind. (In this case, local health authorities are the target 
audience.) Many of the pamphlets address the dissemination of in- 
formation. An example of this is provided in Box 22. 


MAKING EVALUATIONS WORK 


In addition to the issues of communication discussed in the previ- 
ous section, the pursuit of certain goals is more likely to increase the 
utilization of evaluations. A single model for an ideal evaluation of 
environmental health services is, however, impossible to prescribe 
because of the variety of evaluation purposes and the diversity of 
ways to organize evaluations. The objective of this section, there- 
fore, is to highlight favourable conditions and factors that are likely 
to promote better evaluations. 


Favourable conditions for evaluation 

Previous chapters have emphasized that evaluations should follow a 
clear plan and rely on a mixture of data, methods and analyses to 
produce a balanced product. Although each evaluation will have its 
own characteristics, several conditions (or goals) can be defined to 
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Box 22. Ten guidelines for communicating electromagnetic 


field (EMF) issues to a concerned public 





10. 


Never patronize the public. 
Treat all public concerns seriously, with sympathy and consideration. 


Be prepared to spend at least 30 minutes on the telephone with a con- 
cerned individual, though queries can often be dealt with more quickly when 
standard information packs can be posted. 


Take time to explain difficult concepts such as threshold effects, safety 
factors and guideline exposure limits. 


Be honest about the reputable scientific studies that initially gave rise to 
public concern. On the other hand, mention only other equally reputable 
studies that have failed to demonstrate any adverse consequences. 


Stress the importance of replicating any important study that has shown 
possible effects. In general, the public appreciates the fact that standards 
and guidelines cannot be changed every time a single study suggests 
some ill effect. 


With regard to concerns about cancer, emphasize that health advisory 
authorities and governments across the world are funding research, as- 
sembling expert groups and issuing statements on their assessment of the 
latest scientific information. 


Explain that none of the statements issued, or any government, national or 
international health advisory body, is suggesting that EMF from power lines, 
phone towers, mobile phones or broadcasting antennas causes cancer. 


Stress that to prove a negative (that is, that “EMF does not cause cancer”) 
is a logical impossibility. The best that can be done is to show that after a 
great deal of exhaustive research, no evidence can be found that impli- 
cates EMF in the induction or progression of cancer in humans. While much 
research has been completed, more is under way, and it will continue until 
health advisory groups are finally reassured and can advise the public that 
no evidence has been found that links EMF with cancer in humans. 


In dealing with the public, it is useful to provide a summary of the information 
above, together with copies of relevant statements by various national and 
international health advisory authorities. 





Source: WHO Regional Office for Europe (7). 


foster useful evaluations (Box 23). Though these conditions may 
overlap and influence each other, it is unlikely that they will all be 
achieved simultaneously. Evaluators must prioritize these proper- 
ties based on the circumstances, balancing what is desirable with 
what is feasible. Each of the conditions is described more fully in 


the sections that follow. 
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Box 23. Favourable conditions for evaluation 





Ideally evaluations will be arranged so that: 

e they are an integral part of the management cycle; 

they coincide with the expectations and needs of direct stakeholders; 
they deliver reliable, valid, useful and understandable results; 

they are efficient, not only financially but also with respect to other re- 
sources, such as personnel; 

they are both flexible and action-oriented; 

they are completed in a timely fashion; 

they do not interfere with daily routine in environmental health services; and 
they are acceptable to evaluated individuals and stakeholders. 








Integrating evaluation into the management cycle 

Integration means that the evaluation should not be an isolated event. 
External evaluations, for instance, could yield insights that might 
otherwise be overlooked. These insights are more useful when they 
are linked to policy formulation and implementation. Integrating 
evaluation into the management cycle plays a major role in achiev- 
ing effective results. Integration offers three benefits. First, evalua- 
tions that arise from a specific need for information (defined in the 
policy at its implementation stage) are more likely to be put to a 
specific use. Second, strong linkage to the policy formulation stage 
may increase the relevance and usefulness of the results for future 
policies. Third, integrated evaluations are more likely to consume 
fewer additional resources. 


Meeting the needs and expectations of stakeholders 
Environmental health services function among a wide range of or- 
ganizations and stakeholders. Together these groups form what might 
be called the social and political context of evaluation, which af- 
fects the likelihood that an evaluation will be used (6). The social 
context is particularly important, and a wide range of interested and 
involved parties (stakeholders) can be identified. Examples are pro- 
vided in the following list. 


e Decision-makers. These include both commissioners of an 
evaluation and potential users of the results. 


e Funding agencies. These include agencies involved in fund- 
ing specific environmental health service activities and agen- 
cies that might have funded the evaluation itself. 
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e Target groups. These include individuals involved in (or af- 
fected by) the service, programme, activity or policy being 
evaluated (for example, a high risk group). 


¢ Environmental health service managers. These managers may 
play several important roles as commissioners, evaluators and 
users. 


e Environmental health service staff. These people (within an 
environmental health service) may also be evaluated and will, 
of course, have specific expectations and needs with respect 
to the evaluation. 


e Evaluators. These include evaluators themselves as 
stakeholders, especially when they are also users of evalua- 
tion results (for example, management), and those judging 
the evaluation on its methodological strengths. 


e Others. These include technical specialists and other services 
and agencies with similar goals. 


All these stakeholders may be present during the course of an 
evaluation. They have different interests in the evaluation process 
and its results. In order to secure the use of evaluation results, it is 
important to keep the evaluation process open to two-way commu- 
nication with stakeholders — to allow the introduction of new infor- 
mation. Proper planning and regular communication channels are 
essential. More importantly, knowing the needs, expectations and 
preferences of stakeholders can help provide results that will be used. 


Evaluators often have to find a balance between what stakeholders 
view as useful results and what can be done within an evaluation. It 
is impossible to take all views and expectations into account; often 
there are conflicting interests. Among the strategies that can help to 
reduce conflict are the practices of allowing stakeholders time to 
comment, documenting their concerns and responding actively to 
those concerns. Maintaining adequate communication with relevant 
stakeholders is also vital, before, during and after an evaluation. 
Sometimes it is beneficial to plan meetings during the evaluation 
process, so that stakeholders can review the progress and evaluators 
can fine tune their activities (but this is not always feasible). 


134 ENVIRONMENTAL HEALTH SERVICES IN EUROPE 5 


Formulating reliable, valid, useful and understandable 
results 

The validity and reliability of evaluation results are an obvious goal. 
This entire book is directed towards enhancing the validity, reliability 
and credibility of results. Chapter 4, in particular, discusses these con- 
cepts in more detail, as the different methods of evaluation call for dif- 
ferent approaches to ensure and maintain validity and reliability. In ad- 
dition, it is important to emphasize that, although the exact mixture of 
skills necessary for evaluation will vary, evaluators must possess a ba- 
sic level of competence in the techniques of evaluation in order to ob- 
tain valid results. Additional training in communication techniques or 
management theory and practice may also be required. 


Presenting results in a way that is both understandable and useful 
to stakeholders is also a central theme of this book. Highly technical 
results that explain the scientific merits of an evaluation may not, 
for example, be appropriate for all audiences. (See also the sections 
on general dissemination issues, page 127, and communication and 
user involvement, page 137.) 


Promoting efficiency 

As is the case for environmental health services, evaluations them- 
selves need to be carried out efficiently. Ideally, inputs and outputs 
of an evaluation should be balanced, and benefits should outweigh 
the costs involved. 


Evaluations sometimes lack support because they are considered 
an unwarranted additional cost or effort. Evaluators and commis- 
sioners of an evaluation (for example, the management), therefore 
need to promote the efficiency of the evaluation process, before 
and during evaluation activities. This might be done by clearly out- 
lining costs in the evaluation plan and by close cooperation between 
evaluators and commissioners. 


Though financial considerations are essential for efficiency, other 
inputs (such as commitment of personnel) and outputs (such as or- 
ganizational learning and improved relationships with stakeholders) 
are also relevant. Efficiency is often not just a matter of comparing 
numbers, which may be extremely difficult, but is rather a matter of 
instinct, good project management and communication. 
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Keeping evaluations flexible and action-oriented 

In order for evaluations to be successful, they need to be flexible 
and action oriented. Action orientation and integration of an evalu- 
ation in the management process go together; they are complemen- 
tary concepts, as are flexibility and efficiency. Paying specific at- 
tention to planning is essential for developing an evaluation that is 
flexible and aimed at delivering useful results. Flexibility can be 
promoted in evaluation plans that are adaptable and open to informal 
feedback with stakeholders. Orientation towards action can be encour- 
aged through a clear outline of the need and use for evaluation results. 


Producing timely results 

One constraint that evaluators must address is timeliness. An evalu- 
ator may be pressured to deliver fast results, but faster is not neces- 
sarily better. On the other hand, evaluators need to deliver results 
before they become obsolete. In certain planning processes, such as 
the yearly budget cycle, there are specific periods when new infor- 
mation is particularly useful. Timing evaluations so that results are 
disseminated before or during these periods not only fosters the pro- 
duction of useful results but also helps to minimize conflicts be- 
tween evaluators and users. 


Minimizing disruptions to service delivery 

Evaluations can potentially disrupt the daily routine of environmen- 
tal health service delivery. Such interference may make staff mem- 
bers reluctant to cooperate, which can undermine the efficiency and 
effectiveness of the evaluation. Involving staff as evaluation part- 
ners and showing them what benefits will be gained from the evalu- 
ation may help to minimize negative impacts on daily routines. 


Maximizing acceptance of results 

Broad acceptance of the evaluation is also important. To the extent 
possible, efforts should be made to ensure that evaluation activities 
and results are acceptable both to those being evaluated and to the 
broader stakeholder audience. This means that evaluations should 
avoid judging those being evaluated on a personal level. When as- 
sessing service staff performance, it is necessary to develop a full 
understanding of the pressures and obligations under which staff 
work, and to reassure them that the evaluation is intended to be in- 
formative rather than judgemental (see also Chapter 1). 
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Stakeholder acceptance is also essential. Evaluators need to be 
aware of stakeholders’ needs and expectations, and to work to se- 
cure their support throughout the entire evaluation process. Striving 
to make evaluations relevant to stakeholders helps ensure accept- 
ance. (See also the section on meeting the needs and expectations of 
stakeholders, page 132.) 


Core evaluation factors 

Several core factors contribute to an evaluator’s ability to promote 
the goals listed in Box 3 (page 39), including financial resources, 
instrumental resources, evaluation staff, communication and plan- 
ning. Paying close attention to these factors improves the chances 
that evaluation results will be put to good use (7). 


Financial resources 

Sufficient financial resources must be available to conduct evalua- 
tions. Evaluators, however, need to avoid diverting them from other 
service activities. Also, sufficient resources are needed to ensure 
that evaluations remain flexible and open to new problems and 
changing contexts. Environmental health service managers may as 
a standard practice want to consider reserving a small amount of the 
service budget for evaluation activities. 


Instrumental resources 

Specialized equipment required for collecting and storing data and 
information are often referred to as instrumental resources. The term 
covers a wide range of tools, including the basic computer hardware 
and software found in offices, complex environmental sampling 
systems used in the field, analytical instruments found in a labora- 
tory, and even audio and video recording equipment used in inter- 
views and focus groups. Instrumental resources must be of suffi- 
cient quality to supply the evaluators with valid and reliable data. In 
some cases, it may be necessary to acquire new equipment, but in 
other cases pre-existing instruments may suffice. 


The importance of instruments varies depending on the purpose 
of the evaluation and the information needed. Though environmen- 
tal monitoring is impossible without sufficient instruments (such as 
sophisticated sampling equipment and laboratory analysers) these 
tools may be less important to an evaluation of, for example, the 
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relevance of a specific environmental health service activity to the 
public. Computer resources (hardware and software) are usually re- 
quired for every type of evaluation — not only for data processing 
and analysis but also for high-quality compilation and presentation 
of results. 


Evaluation staff 

The personal qualifications of the evaluators also have an influence 
on the way an evaluation is received by stakeholders. Internal and 
external evaluators alike must have: 


e a basic knowledge of evaluation and of the specifics of envi- 
ronmental health and environmental health services; 


e the necessary evaluation skills; and 


e the motivation to perform the evaluation efficiently, effectively 
and equitably. 


The skills and commitment of evaluators are important for main- 
taining flexibility, developing good relations with stakeholders and 
integrating an evaluation with the overall management cycle. 
Multidisciplinary training in relevant environmental health services 
is essential, and communication and management skills must be 
strong in order to interact with interested parties and to cope with 
external pressures (8). 


Communication and user involvement 

The role of communication (both internal and external) in the evalu- 
ation process cannot be overemphasized. Internal communication is 
the exchange of information between members of the evaluation 
team and between the team and the staff evaluated. It is essential for 
securing an open exchange of ideas and new views. External com- 
munication is the exchange of information between the evaluation 
team and stakeholders. Periodic external interactions help evalua- 
tors remain current on the changing service context, evolving per- 
ceptions and new data. Building opportunities for evaluators and 
potential users to exchange information, ideas, needs and expecta- 
tions is recommended. Close contact with users generally improves 
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the utilization of results, making user participation in the design and 
execution of evaluations even more valuable. 


Planning 

Evaluation needs careful planning to provide a practical guideline 
for day-to-day actions. The evaluation plan outlined in Chapter 1 
shows how planning can develop (see Fig. 4 and Box 1). Planning 
helps to ensure that evaluations are integrated into the management 
process and that results are delivered when managers and decision- 
makers need them for developing policy. Planning also keeps evalu- 
ation expenses reasonable by clearly defining tasks, and the costs 
and personnel requirements associated with them. 


Box 24 contains a list of questions that may help make evalua- 
tions more useful. 


CONCLUSION 


An evaluation of environmental health services is not an easy task. 
There is no single recipe for a successful evaluation, because each 
must be developed based on the specific service being evaluated. 
Different environmental health services have different models that 
govern their development and delivery. Evaluation can and should 
interact with these models to improve the design and implementa- 
tion of the services. Making evaluations work is partly an instinctive 
process, perhaps more art than science. The ability to weigh the nec- 
essary objectives and limitations and to strike the right balance is a 
skill that comes only with practice. Depending on the circumstances, 
the different approaches introduced in this book will be more or less 
relevant and feasible. 


Nevertheless, evaluation is an essential component of improving 
the management and delivery of environmental health services across 
the European Region. The necessary time and resources should be 
allocated to make evaluation an integral part of the life cycle of environ- 
mental health service delivery. The Helsinki and London conferences 
have highlighted major challenges faced by environmental health ser- 
vice providers. Meeting these challenges remains an important priority 
for WHO, but will be impossible without commitment to evaluation. 
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Box 24. Questions to help make evaluations more useful 





The following questions are not intended to be a comprehensive check-list, but 
instead are intended to provide a taste of the issues that should be taken into 
account. 


General 

What are the most important qualities this specific evaluation should have? 

What factors contribute to securing these qualities? 

What is the relevance of the evaluation to decision-making and policy formulation? 


Integration 

Is the purpose of the evaluation relevant to decision-making or policy formulation? 
Is the evaluation activity part of a larger process of policy development? 

Do direct stakeholders view the evaluation as an integral component of service 
activity? 

Is there a balance between using existing resources (staff, finance, instruments) 
and potentially new insights that can be provided by “outsiders”? 


Stakeholders 

Who are the main stakeholders and what role do they play? 

How are their preferences dealt with in the evaluation process? 

How do you deal with conflicting interests of stakeholders? 

Is there a line of communication between evaluators and potential users? 

Is communication aimed at different stakeholders or “audiences”? 

Does the evaluation plan or procedure allow for communication with stakeholders? 


Validity, reliability and understandability 

How are validity and reliability assured in the evaluation? 

Does the presentation of results include recommendations for action? 

How are results presented and communicated? 

What is the message to be transmitted? 

Who is to receive this message? 

What specific requirements are necessary to make the message understandable? 
What media are used to disseminate results? (See also the first section in this 
chapter.) 


Efficiency 

How much will this evaluation cost? 

What personnel requirements are needed? 

What benefits can be expected? 

How much additional work is expected from service staff? 
How is efficiency promoted and safeguarded? 


Flexibility and orientation towards action 

Does the evaluation plan leave room for new situations or a changing context? 
How are new situations (such as lack of data) or ideas handled within the evalua- 
tion framework? 

Are evaluators skilled or equipped to handle new situations? 

How are results put into useful information for stakeholders? 

In what time frame are results to be delivered and is this feasible? 


Timely results, daily routine and acceptance 

How might evaluation activities interfere with the daily routine within an environmen- 
tal health service? 

If the possibility for interference exists how can it be prevented? 

When do decision-makers need results of upcoming evaluations? 

Are evaluations and evaluation activities acceptable to stakeholders, especially to 
the people being evaluated? 
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6 


Case Studies of Evaluation 
in Practice 


This chapter provides five case studies of evaluations that were carried 
out in the European Region. Though each case is discussed in detail, 
the collection of cases does not give a representative picture of all evalu- 
ation practices in the Region. Instead, these cases are intended to pro- 
vide the reader with concrete examples with which to compare the theo- 
ries, guidelines and concepts described in previous chapters. 


Each of the five case studies in this chapter includes a brief introduc- 
tion and discussion of the evaluation steps outlined in Fig. 4. The proce- 
dures used to secure support, to define the purpose, subject, scope and 
process (methodology), and to produce and disseminate the results are 
reviewed and discussed. The main features of each evaluation are pre- 
sented in Table 9. 


Member States have expressed an interest in sharing their evaluation 
experiences further. Professional networks, such as the Healthy Cities 
network, have been quite useful for sharing information and learning 
from peers. Should an active information network be established among 
environmental heath professionals, additional examples of good evalu- 
ation practices could easily be incorporated into it. 


THE PROVISION OF AIR POLLUTION ADVICE BY LOCAL AUTHORITIES 


Air quality has been an important public health concern in the United 
Kingdom for at least 100 years. The law recommends that local 
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Table 9. Summary of evaluation case studies 








Case Commissioning Evaluating Subject Tool(s) used 
organization organization 
Air pollution Department of Department of Adequacy and Questionnaire 
advice by local Health Health process of air and database 
authorities in the pollution advice 
United Kingdom given by local 
authorities 
Assessment of | WHO; Health WHO team Process and Site visits and 
environmental and Welfare efficiency of interviews 
monitoring in Canada environmental 
Estonia monitoring 


National Board 
of Health and 
Welfare 


Assessment of 
environmental 
health services 
in Sweden 


Assessment of | Consultation 


National Board 
of Health and 
Welfare 


Consultation 


Efficiency and 
effectiveness of 
environmental 
health service 
action 


Output, process, 


Questionnaires, 
visits and 
interviews, plus 
seminars and 
conferences 


Registration 


an environ- service service efficiency, system and 
mental management management relevance and software 
health and staff effectiveness of 

consultation the service 

service in 

Germany 

Use of Audit Auditors; data Efficiency, Performance 
performance Commission collection also effectiveness indicators 
indicators in the by local and quality of 

United Kingdom authorities services by local 


authorities 





authorities provide health advice to the public when certain levels 
of pollution are exceeded. Provision of such advice by local au- 
thorities, however, has not occurred on a large scale. In dealing with 
air quality, local authorities have developed close ties to the Depart- 
ment of the Environment but the relationship with the Department 
of Health remains fairly undefined. In December 1993, the Depart- 
ment of Health commissioned an evaluation of the provision by lo- 
cal authorities to the public of advice on the health effects of air 
pollution. In brief, survey results show that while the majority (90%) 
of local authorities monitor the quality of the air, only a few (10%) 
distribute advice on the health effects of the pollution. The reason 
stated for not providing advice was that the authorities did not know 
what to say. 
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Step-by-step analysis 

The evaluation was pursued under the authority of the Department of 
Health, but no mention of securing additional political support was made 
in the available documentation (C. Hall, unpublished data, 1994). The 
Department commissioned the evaluation to determine the extent of the 
knowledge “gap” between advice provided and advice requested (or 
required). The subject was the provision of air pollution advice services 
by local authorities in the United Kingdom. More specifically, the scope 
was narrowed to the adequacy of air pollution advice services and the 
process through which they are provided. The Department, as coordi- 
nator of the study, envisaged that the results would aid policy develop- 
ment in the area of air pollution health advisories. The benefits of the 
evaluation would not only go to the public, who should ultimately re- 
ceive better advice on how to deal with periods of high pollution, but 
also to the professionals working in the service itself, because a national 
strategy for improving the overall policy would be developed. 


The study was carried out once, over a total period of about 
6 months. The method consisted of a survey of air pollution moni- 
toring and advice practices. It was sent to 364 district, metropolitan 
and county councils in December 1993. The survey requested in- 
formation on monitoring efforts, information base, local collaborat- 
ing partners, health advice, source of health advice and research. A 
total of 17 questions were asked. It was possible to use a short, rela- 
tively simple questionnaire because the scope of the information 
desired was very limited. A statistical database was established in 
the Department of Health to organize the responses. Over 95% 
(348/364) of the authorities returned the survey within the 3-month 
period allotted. Over 89% (312/348) of the responding authorities 
monitored air quality (a breakdown of pollutants monitored was also 
provided) and disseminated their findings to committees of elected 
members, the general public, the media and other local authorities. 
The local authorities also reported using data for planning proposals 
and local health studies. Despite this abundance of air quality data, 
and interest in air quality, only 32 of the responding authorities (less 
than 10%) provided the general public with any sort of health ad- 
vice that was related to the adverse effects of air pollution. 


The survey illustrated that the adequacy of the air pollution ad- 
vice service in local authorities (as of 1993) was insufficient and 
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that the process for its provision was virtually nonexistent. Although 
many authorities did monitor air quality and did provide the results 
to Councillors, most did not associate the data with their health im- 
plications for local communities. Two reasons were cited for this lack 
of association. 


1. Most local authorities appeared to believe that, since it is ex- 
tremely difficult to provide real-time pollution data, providing 
health information would be irrelevant. 


2. Many authorities did not have sufficient access to the neces- 
sary information that linked health effects with air pollution, 
so they could not provide meaningful advice. 


The final report concludes that even if real-time data could not 
be provided, retrospective advice could still be an important factor 
in instigating behavioural change among the population. The report 
encourages the practice of providing health advice. In addition, it 
recommends several activities to improve the overall provision of 
air pollution health advice to the general public, including the fol- 
lowing: 


e prepare a leaflet that summarizes the health advice issued by 
the Advisory Group on the Medical Aspects of Air Pollution 
(Department of Health) and distribute it to all local authori- 
ties; 


e establish and update annually a database of research work un- 
dertaken by local authorities in the field of air quality and ill 
health; and 


e improve links between local authorities and the Department 
of Health, for example, by contributing regular articles to the 
Department’s Bulletin for environmental health officers. 


The report was issued and distributed within the Department of 
Health. Copies were sent to all local authorities in England. An arti- 
cle for publication in a peer-reviewed journal was submitted by the 
authors. Finally, the results of the evaluation were also presented, at 
technical meetings, to representatives of local authorities. Overall, it 
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is still unclear how the results will actively influence air pollution 
advice policy. Real changes in the practices of local authorities, in 
terms of the way pollution advice is provided, remain to be seen. 


Discussion 

This study was an analysis at the national level of local services. It 
worked in part because the scope was small enough to design a 
short, easily completed, straightforward questionnaire. In addition, 
thorough efforts were made to ensure a good response rate (initial 
and follow-up letters included addressed and pre-paid envelopes). 
The purpose, subject, target audience, evaluators, frequency, meth- 
odology and information base were straightforward and well defined. 
Only two aspects, adequacy and process, were evaluated in any de- 
tail. It is important to recognize, however, that this is not necessarily 
a drawback. Having a limited scope enables the work to be com- 
pleted in a short time and with a relatively simple procedure. For 
example, if a cost analysis had been included, the time needed to 
develop and carry out the survey might have been extended. The 
recommendations clearly show how knowledge gained can be ap- 
plied to future policies and decisions. Besides these positive aspects, 
some features of this study could have been improved. Political sup- 
port, for instance, was not specifically sought, but lack of such sup- 
port did not appear to harm the data collection in this case. 


This case study illustrates how an evaluation of a particular ser- 
vice can identify gaps in the provision of that service. In this case, 
an inadequacy was suspected, a survey was formulated to gather 
data, and the results of the survey were used to develop recommen- 
dations for improving the services. 


ASSESSMENT OF ENVIRONMENTAL MONITORING IN ESTONIA 


In December 1992, after Estonia’s independence, the WHO Regional 
Office for Europe initiated a project to assess Estonian environmen- 
tal monitoring. Previous interviews with professionals had suggested 
that many of the monitoring data in Estonia (and other former Soviet 
republics) were of dubious quality. Since the organization, person- 
nel and methods had not changed after the dissolution of the Soviet 
Union, the validity of the data collected at the time was also in doubt. 
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Both intentional and unintentional misrepresentations of the medical 
and environmental information were suspected. For example, publi- 
cation of certain information on radiation-related exposures and ill- 
nesses was supposedly suppressed. In addition, the number of sick days 
was thought to misrepresent the health of the population, because cli- 
ents often benefited by collecting sick leave authorized by the system. 


A particular problem was seen in the environmental monitoring 
data, especially with regard to laboratory analysis. The use of Soviet 
analytical methods was required in all the former Soviet republics, 
but the requirement often prescribed outdated technology and equip- 
ment. Many data sets were not tested or checked for validity by 
other laboratories, and many laboratories operated with insufficient 
supplies to correctly carry out the specified methodology. Health 
and Welfare Canada funded this project, which was managed by the 
WHO Regional Office for Europe, whose aim was essentially to 
evaluate the process and efficiency of environmental monitoring 
practices in Estonia. The entire project lasted about 10 months (from 
December 1992 to September 1993). 


Step-by-step analysis 

Political support was established by close contact between project 
coordinators and relevant personnel in the Ministry of Health. The 
Government of Estonia was represented throughout the project by 
faculty of the Institute of Chemical Physics and Biophysics. The pur- 
pose of the project was to improve the validity and reliability of (and 
thus, the confidence in) environment and health monitoring data in 
Estonia. The aim of this evaluation was to provide a baseline of data 
from which progress in the quality of data collection could be meas- 
ured. One central question to be answered was: “How well is the 
monitoring being done?” The evaluation also assessed the process 
of collecting and reporting environmental data in Estonia. The types 
of sample chosen for study were: air, groundwater, surface water, 
drinking-water and effluents — each for a selected range of organic 
and inorganic compounds. Three specific tasks were set to assess: 


1. analytical methods used; 

2. existing laboratory facilities; and 

3. environmental analytical data produced in Estonia over the 
previous 5 years. 
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An analysis of efficiency was also required in order to determine 
how well the service was delivered. The immediate benefactors of 
this exercise are assumed to be the planners themselves as well as 
other environmental health professionals in the Ministries of Health 
and of the Environment. The project is a one-time situation assess- 
ment. A team of five Estonian environmental health professionals 
and one WHO consultant was identified by the programme funders 
and managers. The Estonian team compiled a background report for 
the project. The WHO consultant then spent a week in Estonia in 
June 1993 gathering further information during site visits to labora- 
tories and through interviews with environmental health profession- 
als. A further review of analytical methods and facilities on each 
target group of chemicals or compounds was also compiled. Finally, 
proposals for improvements and changes were made to the Govern- 
ment of Estonia during the project review, held in September 1993. 


The site visits confirmed earlier reports of inadequate supplies of 
materials used to analyse samples (such as pure carrier gases for gas 
chromatography analysis and quality filter papers). Also, the lack of 
automation for routine analyses, the shortage of funds and the loss 
of traditional supplies were thought to have contributed to ineffi- 
ciency. The laboratories, now, are attempting to improve the quality 
of their services. Most of those visited had adopted international 
standards for calibration. There was, however, very little evidence of 
quality assurance or quality control within the individual laboratories. 


The following conclusions were drawn in the reports (these find- 
ings are from 1993). 


e A substantial amount of environmental analytical data has been 
produced in Estonia, but not all of it is available and not all of 
it is reliable. 


e Many laboratories have only one method of sophisticated 
analysis available to them (owing mainly to financial con- 
straints), and it is particularly difficult to analyse samples with 
low concentrations of contaminants. Only a few laboratories 
have the ability to analyse many different types of sample. 
There is no single laboratory that can analyse all types of pol- 
lution determinant. 
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Shortages of appropriate spare parts, reagents, standards, etc., 
affect the quality of analytical results, even where suitable in- 
struments are available. 


Several standard methods of analysis are permitted for the same 
pollution determinant. Because the results depend on the 
method used, results from different laboratories are likely to 
be incompatible. 


New methods introduced to the laboratories appear to be in- 
sufficiently validated to estimate the accuracy of the analyti- 
cal results produced. 


There is a serious lack of within-laboratory quality control 
and basic documentation of sampling procedures, sample track- 
ing, analytical procedures actually used, etc. 


Retrospective examination of pollution data would be expensive. 
Overall, the monitoring data are unreliable for a variety of 


reasons, and any use of the existing data might lead to invalid 
conclusions. 


The supporting information required for this assessment was mini- 
mal, but adequate. The Estonian team compiled an extensive back- 
ground report that was supplemented by site visits and unstructured 
interviews. The information collected was compiled in a series of 
freestyle prose reports. No additional computer software was re- 
quired. This exercise mainly provided a baseline of information to 
be used in the future by planners and data managers. In September 
1993 a project review was held in Copenhagen, Denmark, with the 
stakeholders in this project. The following recommendations were 
made to the Government of Estonia. 


1. The quality of data provided by different institutes was ac- 


knowledged to be widely diverse. If the data are to be used, 
they should be carefully assessed for accuracy and reliability. 


2. A few selected chemicals, the occurrence of which can be 


properly monitored, should be identified according to three 
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criteria: existence of technical capacities to produce reliable 
and comparable data; relevance to existing problems; and eco- 
nomic acceptability. 


3. One cause of poor data quality has been a lack of clarity in 
distributing responsibilities. Some tasks are duplicated unnec- 
essarily and others are not completed at all. Reorganizing and 
consolidating health and environmental services would im- 
prove this problem. 


4. Intersectorality should be vigilantly practised in reorganizing 
services. Careful attention should be given to procedures that 
allow the exchange of data and information among various 
ministries. Communities and local authorities should also par- 
ticipate in a two-way flow (to and from) of information. 


5. The highest priority for monitoring should be given to the 
most afflicted regions. 


6. The possibility of establishing relevant structures and identi- 
fying funds for the following environmental health needs 
should be pursued: 


e training personnel; 

* promoting quality control of laboratories; 

* committing laboratories to regular inspection and to au- 
thentication of their procedures; 

* promoting the commitment of management to improving 
quality control; 

e elaborating standard methods; and 

e participating in international programmes. 


7. Reliable channels for supplying required standards, reagents 
and special materials must be established and maintained. Pro- 
duction of such supplies should be promoted in Estonia. Ad- 
vanced techniques, including automation of analyses, should 
also be promoted. 


8. Sound validation of methods should be performed by inde- 
pendent referees and by the laboratories themselves. 
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Several documents were written as a result of this project (/,2). 
The recommendations were submitted to the Government of Esto- 
nia, and the formal project review was archived in the Regional Of- 
fice bibliographic database. 


Discussion 

This project was not designed specifically as an evaluation of Esto- 
nia’s environmental monitoring procedures. Project planners were 
responding to a perceived need to assess the data quality in Estonia. 
The project has been used here as a case study, because all the ma- 
jor evaluation steps occurred. The positive aspects of this case study 
include a clearly defined scope and purpose, a frequency that corre- 
sponds to the specified goals and a thorough methodology appro- 
priate to the purpose. The unstructured nature of the interviews al- 
lowed compilation of diverse information, and may have helped to 
identify new problem areas and collect information about them. Also, 
sufficient situation analysis, adequate information support and good 
application of knowledge were visible in this case. 


Though the overall process was positive, there are some points 
of criticism. Political support could have been better organized be- 
fore the site visit. The Ministry of the Environment was apparently 
not aware of the project and was reluctant to release data to the project 
coordinators without receiving a letter of explanation. It was not 
possible to procure such a letter in the allotted time. This case illus- 
trates that careful planning and communication are essential. Enough 
time should be allotted for contacting the proper people so that the 
proper information can be made available. Another issue was the 
target audience: exactly who was expected to benefit most from the 
activities was not well defined in project materials. Though the mis- 
sion and information gathered for it were well documented, it is un- 
clear how results were distributed. A meeting with the stakeholders 
took place in September 1993, but additional follow-up and assur- 
ance that the recommendations were taken seriously were not ad- 
dressed in the material available. 


Overall, the project helped to define the status of environmental 
monitoring data quality in 1993, but the project’s effectiveness could 
have been improved by defining more clearly how the conclusions 
and recommendations could be used by the Government of Estonia. 
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ASSESSMENT OF ENVIRONMENTAL HEALTH SERVICES IN 
SWEDEN 


The National Board of Health and Welfare (NBHW) is a national 
supervisory authority, connected to the Ministry of Health and So- 
cial Affairs in Sweden. The work of the NBHW is based on different 
plans and regulations aimed at protecting human health, in the fields 
of health care, social welfare, infectious disease control and envi- 
ronmental health. The NBHW supervises the evaluation of different 
health services in the 288 municipalities and the 24 County Coun- 
cils in Sweden.° Important features of these evaluations include a 
county-by-county approach and a dialogue-oriented evaluation. 
These features have been used for about 5 years. 


Local municipalities in Sweden are responsible for such services 
as environment and health protection, social welfare and physical 
planning. With only a few exceptions, however, local municipalities 
are not responsible for health care and medical care. Usually they 
have no medical competence within their organization. Medical ex- 
pertise, including environmental medicine, is available at the County 
Councils. One of the reasons for evaluating the environmental health 
activities of local services is the so-called right to self-determination 
of municipalities. This is a prominent policy principle in Sweden. It 
means that the municipalities have the right to organize their work in 
the way they want, as long as the goals described in the different 
plans are achieved. Against this background, national supervision by 
follow-up and evaluation have become more and more important. 


Step-by-step analysis 

The evaluation activities were performed by (and under the author- 
ity of) the NBHW, with the support of the Ministry of Health and 
Social Affairs. One purpose of the project was to evaluate whether 
the goals in the different regulations on health protection were ful- 
filled, and to promote effective utilization of existing resources. 
Another purpose was to promote the development of local and re- 
gional programmes for evaluating environmental health services. 





5 A series of evaluation reports have been published for each county evaluated. With 
one exception (3), they are not translated into English. For further information, contact 
Mr Martin Eriksson, Head of the Environmental Health Section of the NBHW, Sweden. 
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There are 24 counties in Sweden and every county has an average 
of about 10 municipalities. The NBHW visits about 4 counties each 
year. 


The scope of an evaluation usually covers both general and more 
specific questions, such as: 


e Which local and regional objectives and targets are endorsed? 

e How is environmental health action organized? 

e Which resources are available (in terms of finance and staff- 
ing)? 

* How do the municipalities and County Councils cooperate with 
each other? 

e How do the municipalities and County Councils cooperate with 
other involved parties? 

e Which results are documented? 


The typical county evaluation can be described by a number of 
steps. As a first step, the NBHW decides on a theme, such as health 
risks in the indoor environment. By choosing a theme, the scope of 
the evaluation is narrowed to a certain area of interest. The next step 
is to send questionnaires to all municipalities and the County 
Council. After this, some of the municipalities are visited. Pro- 
fessionals and decision-makers are also interviewed. Results from 
interviews sometimes contrast with answers returned in the ques- 
tionnaires. 


The results of questionnaires, visits and interviews are usually 
published in a report. This report also contains comments from the 
NBHW, together with specific issues that need further attention. As 
part of the dialogue-oriented approach, a seminar is arranged for 
discussions with decision-makers and professionals about the re- 
sults of the evaluation and its conclusions. 


When all municipalities in a county have been evaluated, a con- 
ference is arranged with decision-makers, leading politicians and 
the heads of the different services involved. The Director-General of 
the NBHW leads the conference. The NBHW’s overall conclusions 
are presented and discussed and the NBHW addresses questions on 
how important areas can be improved in the future. 
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As a final step, a small follow-up is performed, about 1—2 years 
after the conference, to see if anything has changed or improved. 
An example of a specific evaluation, on health risks in schools, is 
given in Box 25. The evaluation and its results are supposed to ben- 
efit both the NBHW and the local authorities. The NBHW will ulti- 
mately gain a better insight into the environmental health practices 
at the local level, including the evaluation component, while the 
local services themselves have a chance of improving their activities 
by learning from the results. 


Box 25. Health risks in schools in two counties 


During 1994, the NBHW conducted a follow-up of the inspections of school 
premises in two counties. The aims were to study the occurrence of environ- 
mentally related health problems and to examine how schoolchildren’s need for 
a good indoor environment were met. These aims were directed by the munici- 
palities’ responsibility to control the environmental health status in these premises, 
as laid down by health protection legislation, and the County Councils’ overall 
duty to prevent ill health, as laid down by the Health and Medical Service Act. 


The structure (for example, organization and resources), the process (for 
example, priorities, plans, objectives, targets and service activities) and the 
results in two counties and in the municipalities in the counties were described. 
A special question about local and regional evaluation of these services was 
asked. 


Comments on the actual quality of objectives (the structural quality and the 
process quality) were given by the NBHW. Some of the comments follow. 


° The inspection work would benefit from a clearer statement of objectives 
and priorities. The objectives must be more than just visions: they must be 


operative and permit follow-up and evaluation. 


Resources, in terms of competence levels and staffing levels, are too small 
to permit the offices to satisfactorily perform their duty of inspecting educa- 
tional premises. Overall regional resources should be reassessed. The 
possibilities for pooling resources should be considered. 


Many local environment and health departments are self-critical and con- 
sider themselves to be insufficiently aware of the general standard of edu- 
cational premises, especially of the effects premises may have on a pupil’s 
state of health. 


In the municipalities where the inspection work has been most successful, 
inspection has been carried out in the form of a joint inventory project 
conducted with both the Real Estate Department and the school’s gover- 
nors. This approach mobilizes a broader base of experience and know- 
ledge than is available when the environment and health departments carry 
out the inspection by themselves. The effectiveness might be increased 
through county-wide coordination of such projects. 
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Box 25 (contd) 





° The environment and health departments ought to maintain a better level of 
contact with the Occupational Safety Inspectorate, to promote greater ef- 
fectiveness. 


e Of the 35 municipalities evaluated in these counties, only 6 conduct a 
regular follow-up of a school’s indoor environment, and only one of these 
had an established structure for performing follow-up and evaluation. 


The following questions were posed for consideration. 


e How can the County Council’s work in environmental medicine be made 
more region-specific and how can it be given directed applicability so that, 
to a greater extent than at present, it leads to collaboration between munici- 
palities in their efforts to mitigate and assess current problems? 


e How can the region’s work in the area of environmental health be made 
more effective, by coordination of planning and joint resource use? 


e How can a continual supply of region-specific expertise be ensured in the 
area of environmental health? 


e In what way(s) can follow-up and evaluation on schoolchildren’s health 
contribute to increased effectiveness in the local environment and health 
departments’ inspection of educational premises? 


e Who is responsible for strengthening systems that make it possible to 
register environmentally related health problems in children? 








Source: Eriksson et al. (3). 


Discussion 

The dialogue-oriented follow-up and evaluation has proved success- 
ful, and it has initiated processes at the local level. Moreover, the 
approach is appreciated by the local and regional authorities. They 
appreciate that national authorities are coming out “in the field” and 
creating direct personal contact and dialogue. For the NBHW, it of- 
fers a unique opportunity to learn of the current situation at the local 
level, especially as a starting point for different national measures. 


An important purpose of the approach is to initiate local routines 
of follow-up and evaluation of the municipalities’ effectiveness within 
their own services. Very often such an evaluation routine is lacking, 
and local targets are often so general that they are not fit to be a 
starting point for an evaluation. To promote higher quality in the 
local evaluation, the NBHW is currently working with a handbook 
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that translates the relevant ISO standard into an applicable model 
for evaluating local environmental health services. The direct per- 
sonal contact and the dialogue achieved by the interviews and the 
seminars are expected to be important determining factors in a suc- 
cessful evaluation. In addition, the project is significantly important 
in promoting processes that improve environmental health services. 
Another advantage of the project is that it is also relatively cheap. 


ASSESSMENT OF AN ENVIRONMENTAL HEALTH CONSULTATION 
SERVICE 

The Free and Hanseatic City of Hamburg is one of the smallest of 
the 16 German Ldnder (States), but it has a long tradition of innova- 
tive environmental health services. The Division of Health and En- 
vironment is one division in the Office of Health. An environmental 
health consultation service (a clinic) was started in response to a 
publicly perceived need for a centre specializing in the diagnosis 
and treatment of problems that result from exposure to environmen- 
tal pollution. This consultation service was designed as a pilot project, 
and an evaluation component was required from the very begin- 
ning. The evaluation procedure (4) was designed to provide timely 
information on the services to the managers of the programme, and 
to help the managers make decisions about the process and struc- 
ture of the clinic itself. 


Step-by-step analysis 
Accounts of attempts to draw political support for this evaluation 
were not included in the available literature (5). 


The purpose of the evaluation was three-fold: 


1. to show that the concept of an environmental health consulta- 
tion service is transparent and rational; 

2. to help facilitate exchange of experience with other consulta- 
tion services; and 

3. to create a basis for making decisions about the further devel- 
opment of the service. 


Ultimately, the planners hoped to show the usefulness of the in- 
formation system that had been designed and to use the evaluation 
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to plan a better consultation service, both for the future and for other 
localities. 


The subject of the evaluation was to be very broad. The outputs, 
process, efficiency, relevance and effectiveness of the services were 
to be analysed. The workers in the environmental health consulta- 
tion service carried out the activities for the evaluation, and their 
reports were reviewed by the donor groups and by the Health and 
Environment Division of the Office of Health. The ultimate target 
group for the consultation service is people exposed to environmen- 
tal pollution. The target for the evaluation itself, however, was not 
only to help the general public but also to help other medical practi- 
tioners, other consultation service planners and the planners of this 
service, so that better-informed decisions could be made for the fol- 
lowing years. (The consultation service will be evaluated each year.) 


A reporting system was developed to provide information to the 
programme managers. Two types of report are kept on separate cards. 
The first type is the individual patient record, which includes per- 
sonal details (such as name, address and occupation), a brief medi- 
cal history, how the patient heard about the service, the nature of the 
exposure suspected and the problems experienced by the patient. A 
separate set of cards was organized according to the problems en- 
countered. Records traced the actions taken for each problem en- 
countered. Each set of cards was analysed separately with the com- 
puter program Epi-Info.° Staff in the Health and Environment Divi- 
sion of the Office of Health helped the clinicians working in the 
consultation clinic analyse the data and draw conclusions. 


The programme was judged to be a success both in general and 
in financial terms. It evolved, however, a bit differently than first 
envisaged. The main thing to be analysed by the first year’s evalua- 
tion was the evaluation process itself. Originally, the evaluation com- 
ponent was expected to fulfil the need for balance in the project and 
to be an instrument of control and monitoring. But during the 
course of the project, there was a dynamic interaction between 
the direct collection of data and the evaluation component. As a 


° Epi-Info is anon-copyrighted, free software program distributed by the US Centers 
for Disease Prevention and Control and WHO. 
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result, the entire concept of the consultation service changed (see 
also Discussion, page 158). 


The assessment of the evaluation process identified many prob- 
lems that can be solved through good planning. One key lesson 
learned was about the need for minimizing the amount of paper 
work. Another important lesson learned was how to use the evalua- 
tion as a control mechanism. If, however, the data are to be useful, 
they must be analysed regularly and applied where appropriate. By 
looking at the number of people who called or came to the clinic, 
the programme planners judged the service to be relevant to the 
needs of the community and argued not only that the programme 
should be continued, but also that it should be implemented in other 
locations. 


Information collected by the consultation service staff, who asked 
structured questions of their clients, was used to inform the adminis- 
tration how often the public used the consultation services, which 
provided a strong argument to continue and expand this type of 
service. In addition, a clear baseline of information was developed. 
The baseline of information could then be used in comparisons with 
data gathered, for example, after a special media event describing a 
spill or a particular hazard about which people should be concerned. 


The evaluation system was, in part, designed to help planners 
make decisions. In this respect, it served its purpose well. For exam- 
ple, the service was originally intended to provide extensive medi- 
cal examinations of people who suspected they had been exposed 
to something harmful in the environment. But, since most of the 
people who contacted the service wanted only general information, 
the procedures for collecting information changed. The managers 
realized that collecting detailed socioeconomic and medical histo- 
ries from such callers drained scarce work time that could have been 
used more productively. In the end, procedures were changed so 
that only minimal background information was collected from call- 
ers with general questions, and more detail was collected from those 
receiving more extensive medical examinations. 


The evaluation was also to be used to justify the continuation 
and expansion of the project to other districts. Thus, disseminating 
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the findings of the evaluation was important. Much of the informa- 
tion gained by implementing the service and evaluating it has been 
published in peer-reviewed journals. 


Discussion 

As this is the only case study described in this publication that was 
specified as an evaluation from the very beginning, most of the steps 
recommended here were carried out. Especially important were the 
ideas to use the information gained throughout the process to im- 
prove the process itself and to make decisions about the procedures 
and activities in the future. In this case, the application of the infor- 
mation gathered is very well developed, because one of the stated 
purposes of the evaluation was to provide a tool that could be used 
by management to improve the service. Unfortunately, there were 
also some negative results, despite efforts to avoid having only a 
positive or negative connotation. The consultation service staff felt 
that they, personally, had been judged positively or negatively on 
their work. The programme managers continued to stress the impor- 
tance of reporting on the work itself, not just the data collected from 
patients. 


In summary, this case study illustrates an evaluation as a 
predefined component of a service. It shows that even with good 
planning, evaluation and monitoring systems can indicate areas for 
improvement. 


PERFORMANCE INDICATORS IN THE UNITED KINGDOM 


The Audit Commission for Local Authorities and the National Health 
Service in England and Wales has been established as the watchdog 
for fiscal accountability at the local level. The Commission promotes 
quality, economy, efficiency and effectiveness in local authorities 
by appointing auditors for authorities, setting and maintaining stand- 
ards for auditing, and investigating and reporting on the impact of 
local authorities. In 1992, the Local Government Act was passed 
requiring the Audit Commission to develop a set of indicators for 
services delivered by local authorities. After collaborating with many 
representatives from relevant sectors and local authorities themselves, 
a list of performance indicators to be addressed in the first year 
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(1993/1994) was developed. The indicators are part of a wider gov- 
ernment initiative to make public services more open and account- 
able to the public: the so-called Citizen’s Charter. The list of indica- 
tors was expanded for the second year of the programme (1994/ 
1995). Many of these indicators address environmental health ser- 
vices that are delivered mainly at the local level (6—/4). 


The Audit Commission divides the indicators into four different 
categories: performance, cost, quality and background information, 
and provides the following description of each. 


e Performance indicators include the concepts of efficiency and 
effectiveness. Some of these indicators are more prescriptive 
than others. At one extreme, the Commission requires only 
that something be measured and some target be set, giving 
authorities more autonomy; at the other extreme, specific things 
have to be measured and targets are set by the Commission. 


e Cost indicators are meant to measure value. Generally, these 
are expressed as price per unit of service, or as cost per per- 
son. The Commission recognizes that it is important not to 
assume that high costs mean low value. These indicators are 
used alongside performance indicators to determine overall 
value for money. 


e The Audit Commission considers quality indicators to be best 
measured by groups of indicators. The Commission has de- 
veloped check-lists of the components that services should 
have. Similarly, no one service is represented by only one in- 
dicator. 


e Finally, indicators that provide background information help 
to place services in a social, economic, political or historical 
context. These indicators (for example, how much a service is 
used or what the population using a service is like) help link 
outputs and outcomes of services to the amount of work re- 
quired to provide them. 


In the first year of the programme (1993/1994), the Audit Com- 
mission used only a few environmental health indicators. The number 
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of indicators collected will increase gradually every year, so envi- 
ronmental health indicators are likely to improve in both number 
and quality. Box 26 shows the topics of the indicators for the first 
two years and beyond. 


Step-by-step analysis 

In 1983, the Government made local authority auditing (a tradition 
since 1846) the responsibility of an independent body, the Audit 
Commission. In 1992, the Commission’s duties were expanded, and 
it began to issue directives for local authorities to publish compara- 
tive performance indicators. The Government supported these ac- 
tivities, although the Commission operated independently as a non- 
departmental public body. The purpose of the Citizen’s Charter was 
to ensure that authorities were accountable to their public. As a con- 
sequence, the comparisons made between authorities had to be ac- 
cessible to the public. 


There is a difference between the general goal of the Charter and the 


more specific and operational goals of environmental health indicators 
of local authorities. The subject has been clearly defined as the services 


Box 26. Citizen’s Charter indicators 





The environmental health indicators are shown in italic. 


Time frame Topics 


First year (1993/1994) Council tax and housing benefit, council tax collec- 
tion, development control and local plans, educa- 
tion, fire, housing, refuse collection, police, public 
libraries, social services, waste disposal 


Second year (1994/1995) Environmental health, highways and rural footpaths, 
leisure parks and open spaces, public toilets, street 
cleaning, street lighting, trading standards 


Not yet covered Building control, coastal protection, county parks 
and tenanted farms, economic development, licens- 
ing, museums and art galleries, strategic planning, 
tourism 


a Housing has not been included as an environmental health topic because the 
housing indicators selected thus far are principally financial in nature and do not 
relate to environmental health. 








Source: Audit Commission (10). 
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provided by local authorities in the United Kingdom. This is an ex- 
tremely broad subject, but was more narrowly defined for the first 
year. In subsequent years more topics to be evaluated were added, 
widening the scope of the evaluation. The local authorities conducted 
the evaluation and gathering of data by themselves. To guarantee 
that systems for gathering data were of sufficient quality, agents from 
the Audit Commission were also involved in the evaluation work, 
though indirectly. The final evaluation results were intended for pub- 
lic consumption. It was envisaged that the public would use the indi- 
cators to judge the performance of the local authorities. Politicians 
were also a target audience. The information was to be collected and 
published on an annual basis. The Audit Commission has devel- 
oped sets of indicators to measure the quality, economy, efficiency 
and effectiveness of services provided by local authorities. A wide 
range of interested parties was consulted over a period of several 
years in order to create the indicators. Trained auditors visited the 
local authorities gathering the required information. Only four sets 
of 1994/1995 indicators that are particularly relevant for evaluating 
environmental health are presented here: dealing with the public, 
refuse collection, waste disposal and environmental health (Box 27). 


The stated purpose of the indicators developed by the Audit Com- 
mission was to ensure the quality, economy, efficiency and effec- 
tiveness of local authorities. The relevance and adequacy of the ser- 
vices were not stated priorities of the Audit Commission or the Citi- 
zen’s Charter. Figures were published in the local press, and drew 
national press interest. Information sources were clearly defined and 
were to be obtained through local authority statistics. The Authori- 
ties were told in advance what data were to be collected in order that 
they could provide them. The Commission’s agents received detailed 
instructions on collecting information. For each indicator topic there 
were several figures collected. The Citizen’s Charter requires that 
results be published in local newspapers. National newspapers were 
to publish aggregate results and comparisons. Several studies were 
done to determine the best way of presenting the information to the 
public, and advice on presentation of material was disseminated to 
the local authorities. 


The Citizen’s Charter indicators were designed to measure how 
well local authorities in the United Kingdom were doing their job, 
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Box 27. Relevant performance indicators 


Dealing with the public 


Answering the phone 
What is the authority’s target for 
answering calls? 
How was performance moni- 
tored? 
What was the performance against 
the targets? 
Answering letters 
What is the authority’s target for 
answering letters? 
How was performance moni- 
tored? 
What was the performance against 
the targets? 
Complaints to local authority om- 
budsman 
How many complaints were con- 
sidered by the ombudsman? 
How many were classified as: 
(i) local settlement; (ii) maladmin- 
istration with no injustice; (iii) mal- 
administration with injustice? 
Handling complaints 
What is the authority’s definition of 
a complaint? 
What are the answers to the fol- 
lowing questions? 
Is there an up-to-date written 
policy available to the public? 
Does it contain information on 
the procedure for making 
complaints? 
Does it clearly describe allo- 
cation of responsibility and ac- 
tions required? 
Does it contain time limits and 
targets? 
Does it specify that when lim- 
its are exceeded complain- 
ants must receive notice? 
Does it specify that written 
complaints must receive a 
written response? 
Is there a follow-up procedure 
if acomplainant is not satisfied? 
Does the authority have a writ- 
ten policy on remedies? 
Is there a system for review- 
ing the causes of complaints? 
Does the authority publish a 
report on complaints that is 
available to the public? 


5. Access to buildings 


What is the number of the authori- 
ty’s buildings (offices) open to the 
public? 

What is the number of these build- 
ings in which all public areas are 
accessible to disabled persons? 


Refuse collection 


1. 


The service provided 

Does the authority provide contain- 
ers for household waste? 

Does the authority provide wheeled 
bins for household waste? 

Is household waste collected from 
the back door of domestic proper- 
ties? 

Is garden waste collected? 

Is garden waste collected free of 
charge? 

Are appointments given for collec- 
tion of bulky waste? 

Is bulky waste collected free of 
charge? 

Are recyclable materials collected 
separately from household waste? 
Is a direct dial telephone service 
available 8 hours per working day 
and is there a phone answering 
service that takes messages of 
complaint at all other times? 
Reliability 

What is the authority’s target for re- 
liability of household waste collec- 
tion service? 

What was the performance against 
the targets? 

What is the authority’s target for rec- 
tifying errors? 

What was the performance against 
the target (rectifying error)? 
Recycling 

How many tonnes of household 
waste were collected? 

What was the percentage of house- 
hold waste recycled? 

Expenditure 

How many households were in- 
cluded? 

What was the net cost per house- 
hold? 


Waste disposal 


1. 


How much household waste was 
received? 
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Box 27 (contd) 


What is the percentage of 
household waste that was: 
recycled? 

incinerated with recovery of heat 
and power? 

incinerated without the recovery 
of heat and power? 

disposed of in other ways? 
What was the net cost per tonne 
of household waste received? 


Environmental health (the local 
environment) 


1. 
2. 


How clean are the streets? 
What is the percentage of high- 
ways that were: 

of a high standard of cleanliness? 
of an acceptable standard of 
cleanliness? 

What was the method of inspec- 
tion used to monitor item 2? 
What was the average time taken 
to remove fly-tips? 

How many public conveniences 
were there? 

What was the percentage of these 
providing: 

(i) access for disabled people? 
(ii) baby changing facilities? 
What is the target time for re- 
sponse to complaints? 


What is the target time set by the 
authority for responding to environ- 
mental health complaints and re- 
quests for services? 

What is the percentage of re- 
sponses provided within the target 
times? 

What is the scope of food inspec- 
tion? 

How many food premises are 
there? 

What was the percentage of such 
premises due to be inspected that 
were inspected? 

How timely was the response to 
consumer protection complaints? 
What is the target time set by the 
authority for responding to con- 
sumer protection complaints and 
requests for services? 

What was the percentage of re- 
sponses provided within the target 
times? 

What is the scope of the inspection 
of premises covered by consumer 
protection legislation? 

How many premises were covered 
by consumer protection legislation? 
What was the average number of 
visits per high and medium risk 
premises? 
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thereby showing the general public whether their money was well 
spent. One can presume that if citizens expressed dissatisfaction, 
the authorities would attempt to improve their services. But in the 
literature reviewed, the Audit Commission did not provide guidance 
on how local authorities could use the indicators to improve ser- 
vices, only guidance for citizens on how to interpret the indicators. 


The Commission went through a review process for the first set 
of indicators, and a few changes and additions were made for the 
second year. It is important to note that the Audit Commission does 
have guidelines on many types of local service, including the Na- 
tional Health Service and the Environmental Health Services. But 
there is no direct evidence that specific recommendations on the 
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best course of action in certain situations were provided to the local 
authorities. 


The Audit Commission looked towards the future. This was clearly 
a long-term project and a large undertaking. Each year new indica- 
tors were to be added. In addition, the indicators themselves were 
reviewed and revised as appropriate. For example, from the 1993/ 
1994 period to the 1994/1995 period, language was clarified, a few 
indicators were deleted and many new indicators and topics were 
added. 


The Citizen’s Charter mandated that the results be published in 
local newspapers. The Audit Commission had also developed a docu- 
ment entitled Watching their figures. A guide to the Citizens Charter 
indicators (14), which was targeted at helping lay citizens under- 
stand the laws and use the results to judge local authorities. The 
indicators, first published in early 1995, are packaged in a two- 
volume set with an additional index. 


Discussion 

The Audit Commission does meet its own goal of developing indi- 
cators that are disseminated to the public, thereby providing the public 
with a mechanism for evaluating the services. The Commission has 
established a concrete set of data against which the public can react. 
In this way, the public may become better incorporated into the de- 
cision-making process — for example, by knowing exactly what ser- 
vices are being provided by the authorities, and at what price. Theo- 
retically, major changes in public consensus on which services should 
be provided might be reflected in the response to the publication of 
the indicators. Though the set of indicators developed by the Audit 
Commission contains good points, it is not flawless. Several areas 
have room for improvement — in particular, with regard to method- 
ology, actual evaluation and formulation of conclusions. 


The methodology could be improved in a number of ways. One 
way to improve it is to provide more detail on how information would 
be supplied. Another way is to improve the methodology used to 
develop the indicators. Most of the indicators do little more than 
count activities that have been deemed important. It seems that few 
actual decisions or conclusions will result from the efforts of the 
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auditing exercise. For the analysis, it must be noted that the indica- 
tors developed reflect poorly the effectiveness of the services. Fur- 
thermore, relevance and adequacy are virtually ignored. An overall 
review of environmental health services should include these sub- 
jects. In addition, there are voids in the methodology: there is no 
mechanism for asking whether an alternative service could have been 
provided; there is a lack of a clear explanation that links the infor- 
mation provided by the indicators with the efficiency, economy, 
quality and effectiveness of the services; and there is no guidance 
on using this information to improve future services. 


The indicators encompass a narrow view of environmental health 
services. The areas receiving major attention are food inspection, 
waste collection, waste disposal, public conveniences, response to 
complaints and housing. No indicators, however, have been devel- 
oped for water services, monitoring pollution (air, water, soil, noise, 
light) and occupational health. Also, the housing indicators, though 
not discussed in depth, generally concentrate on finance rather than 
environmental health. Such a narrow set of indicators may result in 
local authorities ignoring services or aspects of services that are not 
specified by the Commission. This may be extremely problematic 
when specific local problems that should be addressed by the local 
authority are ignored because no indicators are mandated to be pub- 
lished in the newspaper for that particular service. Put another 
way, too much centralized control of local services may violate 
the concept of subsidiarity (government authority should reside 
at the level that is most appropriate to the specific goals of the 
policies). 


The intended use of the indicators is also too limited. Fiscal ac- 
countability to the public is clearly a relevant goal, but it does not go 
far enough. The public needs indicators so it can make decisions 
about future services. Indicators could also be used more effectively 
to introduce innovation into the local government process. Though 
the Audit Commission says this is an aim, it does not describe how 
the aim is to be realized. Because politicians have been defined as a 
potential target group, there is a danger that the results of the per- 
formance indicators could be manipulated for personal political gain. 
The Audit Commission’s heavy reliance on statistical and monetary 
figures provides a tool that might be manipulated to such an end. 
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This case study is meant to provide an example of one type of 
evaluation of environmental health services occurring in Europe to- 
day. It is our understanding that the Audit Commission has incorpo- 
rated as many stakeholders as possible into the development of this 
system of performance indicators. To be fair, we have to remember 
that the Audit Commission does not have an intrinsic mandate to 
evaluate environmental health services, and environmental health 
services are by no means the primary reason it has developed these 
performance indicators. Nevertheless, this case study has been in- 
cluded in order to illustrate certain points and to further the debate 
on the development of effective performance indicators with which 
to evaluate public services, including, of course, environmental health 
services. 
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